Fix 8244 -- store timestamps with nanosecond resolution (consistently) (#8337)
# Description
Fix for data ambiguity noted in #8244.
Basic change is to use nanosecond resolution for unix timestamps (stored
in type Int). Previously, a timestamp might have seconds, milliseconds
or nanoseconds, but it turned out there were overlaps in data ranges
between different resolutions, so there wasn't always a unique mapping
back to date/time.
Due to higher precision, the *range* of dates that timestamps can map to
is restricted. Unix timestamps with seconds resolution and 64 bit
storage can cover all dates from the Big Bang to eternity. Timestamps
with seconds resolution and 32 bit storage can only represent dates from
1901-12-13 through 2038-01-19. The nanoseconds resolution and 64 bit
storage used with this fix can represent dates from 1677-09-21T00:12:44
to 2262-04-11T23:47:16, something of a compromise.
# User-Facing Changes
_(List of all changes that impact the user experience here. This helps
us keep track of breaking changes.)_
## `<datetime> | into int`
Converts to nanosecond resolution
```rust
〉date now | into int
1678084730502126846
```
This is the number of non-leap nanoseconds after the unix epoch date:
1970-01-01T00:00:00+00:00.
Conversion fails for dates outside the supported range:
```rust
〉1492-10-12 | into int
Error: nu::shell::incorrect_value
× Incorrect value.
╭─[entry #51:1:1]
1 │ 1492-10-12 | into int
· ────┬───
· ╰── DateTime out of timestamp range 1677-09-21T00:12:43 and 2262-04-11T23:47:16
╰────
```
## `<int> | into datetime`
Can no longer fail or produce incorrect results for any 64-bit input:
```rust
〉0 | into datetime
Thu, 01 Jan 1970 00:00:00 +0000 (53 years ago)
〉"7fffffffffffffff" | into int -r 16 | into datetime
Fri, 11 Apr 2262 23:47:16 +0000 (in 239 years)
〉("7fffffffffffffff" | into int -r 16) * -1 | into datetime
Tue, 21 Sep 1677 00:12:43 +0000 (345 years ago)
```
## `<date> | date to-record` and `<date> | date to-table`
Now both have a `nanosecond` field.
```rust
〉"7fffffffffffffff" | into int -r 16 | into datetime | date to-record
╭────────────┬───────────╮
│ year │ 2262 │
│ month │ 4 │
│ day │ 11 │
│ hour │ 23 │
│ minute │ 47 │
│ second │ 16 │
│ nanosecond │ 854775807 │
│ timezone │ +00:00 │
╰────────────┴───────────╯
〉"7fffffffffffffff" | into int -r 16 | into datetime | date to-table
╭───┬──────┬───────┬─────┬──────┬────────┬────────┬────────────┬──────────╮
│ # │ year │ month │ day │ hour │ minute │ second │ nanosecond │ timezone │
├───┼──────┼───────┼─────┼──────┼────────┼────────┼────────────┼──────────┤
│ 0 │ 2262 │ 4 │ 11 │ 23 │ 47 │ 16 │ 854775807 │ +00:00 │
╰───┴──────┴───────┴─────┴──────┴────────┴────────┴────────────┴──────────╯
```
This change was not mandated by the OP problem, but it is nice to be
able to see the nanosecond bits that were present in Nushell `date` type
all along.
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
2023-03-08 00:02:15 +01:00
|
|
|
use chrono::{DateTime, FixedOffset, NaiveDate, TimeZone};
|
|
|
|
use rstest::rstest;
|
|
|
|
|
2023-07-17 18:43:51 +02:00
|
|
|
use nu_test_support::nu;
|
2020-08-27 07:44:18 +02:00
|
|
|
|
|
|
|
#[test]
|
|
|
|
fn into_int_filesize() {
|
2023-07-17 18:43:51 +02:00
|
|
|
let actual = nu!(r#"
|
2022-02-17 12:40:24 +01:00
|
|
|
echo 1kb | into int | each { |it| $it / 1000 }
|
2023-07-17 18:43:51 +02:00
|
|
|
"#);
|
2021-02-10 03:31:12 +01:00
|
|
|
|
|
|
|
assert!(actual.out.contains('1'));
|
|
|
|
}
|
|
|
|
|
|
|
|
#[test]
|
|
|
|
fn into_int_filesize2() {
|
2023-07-17 18:43:51 +02:00
|
|
|
let actual = nu!(r#"
|
2022-02-17 12:40:24 +01:00
|
|
|
echo 1kib | into int | each { |it| $it / 1024 }
|
2023-07-17 18:43:51 +02:00
|
|
|
"#);
|
2020-08-27 07:44:18 +02:00
|
|
|
|
2020-10-08 23:47:51 +02:00
|
|
|
assert!(actual.out.contains('1'));
|
2020-08-27 07:44:18 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
#[test]
|
|
|
|
fn into_int_int() {
|
2023-07-17 18:43:51 +02:00
|
|
|
let actual = nu!(r#"
|
2022-02-17 12:40:24 +01:00
|
|
|
echo 1024 | into int | each { |it| $it / 1024 }
|
2023-07-17 18:43:51 +02:00
|
|
|
"#);
|
2020-08-27 07:44:18 +02:00
|
|
|
|
2020-10-08 23:47:51 +02:00
|
|
|
assert!(actual.out.contains('1'));
|
2020-08-27 07:44:18 +02:00
|
|
|
}
|
2022-07-03 20:31:50 +02:00
|
|
|
|
|
|
|
#[test]
|
|
|
|
fn into_int_binary() {
|
2023-07-17 18:43:51 +02:00
|
|
|
let actual = nu!(r#"
|
2022-07-03 20:31:50 +02:00
|
|
|
echo 0x[01010101] | into int
|
2023-07-17 18:43:51 +02:00
|
|
|
"#);
|
2022-07-03 20:31:50 +02:00
|
|
|
|
|
|
|
assert!(actual.out.contains("16843009"));
|
|
|
|
}
|
Fix 8244 -- store timestamps with nanosecond resolution (consistently) (#8337)
# Description
Fix for data ambiguity noted in #8244.
Basic change is to use nanosecond resolution for unix timestamps (stored
in type Int). Previously, a timestamp might have seconds, milliseconds
or nanoseconds, but it turned out there were overlaps in data ranges
between different resolutions, so there wasn't always a unique mapping
back to date/time.
Due to higher precision, the *range* of dates that timestamps can map to
is restricted. Unix timestamps with seconds resolution and 64 bit
storage can cover all dates from the Big Bang to eternity. Timestamps
with seconds resolution and 32 bit storage can only represent dates from
1901-12-13 through 2038-01-19. The nanoseconds resolution and 64 bit
storage used with this fix can represent dates from 1677-09-21T00:12:44
to 2262-04-11T23:47:16, something of a compromise.
# User-Facing Changes
_(List of all changes that impact the user experience here. This helps
us keep track of breaking changes.)_
## `<datetime> | into int`
Converts to nanosecond resolution
```rust
〉date now | into int
1678084730502126846
```
This is the number of non-leap nanoseconds after the unix epoch date:
1970-01-01T00:00:00+00:00.
Conversion fails for dates outside the supported range:
```rust
〉1492-10-12 | into int
Error: nu::shell::incorrect_value
× Incorrect value.
╭─[entry #51:1:1]
1 │ 1492-10-12 | into int
· ────┬───
· ╰── DateTime out of timestamp range 1677-09-21T00:12:43 and 2262-04-11T23:47:16
╰────
```
## `<int> | into datetime`
Can no longer fail or produce incorrect results for any 64-bit input:
```rust
〉0 | into datetime
Thu, 01 Jan 1970 00:00:00 +0000 (53 years ago)
〉"7fffffffffffffff" | into int -r 16 | into datetime
Fri, 11 Apr 2262 23:47:16 +0000 (in 239 years)
〉("7fffffffffffffff" | into int -r 16) * -1 | into datetime
Tue, 21 Sep 1677 00:12:43 +0000 (345 years ago)
```
## `<date> | date to-record` and `<date> | date to-table`
Now both have a `nanosecond` field.
```rust
〉"7fffffffffffffff" | into int -r 16 | into datetime | date to-record
╭────────────┬───────────╮
│ year │ 2262 │
│ month │ 4 │
│ day │ 11 │
│ hour │ 23 │
│ minute │ 47 │
│ second │ 16 │
│ nanosecond │ 854775807 │
│ timezone │ +00:00 │
╰────────────┴───────────╯
〉"7fffffffffffffff" | into int -r 16 | into datetime | date to-table
╭───┬──────┬───────┬─────┬──────┬────────┬────────┬────────────┬──────────╮
│ # │ year │ month │ day │ hour │ minute │ second │ nanosecond │ timezone │
├───┼──────┼───────┼─────┼──────┼────────┼────────┼────────────┼──────────┤
│ 0 │ 2262 │ 4 │ 11 │ 23 │ 47 │ 16 │ 854775807 │ +00:00 │
╰───┴──────┴───────┴─────┴──────┴────────┴────────┴────────────┴──────────╯
```
This change was not mandated by the OP problem, but it is nice to be
able to see the nanosecond bits that were present in Nushell `date` type
all along.
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
2023-03-08 00:02:15 +01:00
|
|
|
|
|
|
|
#[test]
|
|
|
|
#[ignore]
|
|
|
|
fn into_int_datetime1() {
|
|
|
|
let dt = DateTime::parse_from_rfc3339("1983-04-13T12:09:14.123456789+00:00");
|
|
|
|
eprintln!("dt debug {:?}", dt);
|
|
|
|
assert_eq!(
|
|
|
|
dt,
|
|
|
|
Ok(FixedOffset::east_opt(0)
|
|
|
|
.unwrap()
|
|
|
|
.from_local_datetime(
|
|
|
|
&NaiveDate::from_ymd_opt(1983, 4, 13)
|
|
|
|
.unwrap()
|
|
|
|
.and_hms_nano_opt(12, 9, 14, 123456789)
|
|
|
|
.unwrap()
|
|
|
|
)
|
|
|
|
.unwrap())
|
|
|
|
);
|
|
|
|
|
|
|
|
let dt_nano = dt.expect("foo").timestamp_nanos();
|
|
|
|
assert_eq!(dt_nano % 1_000_000_000, 123456789);
|
|
|
|
}
|
|
|
|
|
|
|
|
#[rstest]
|
|
|
|
#[case("1983-04-13T12:09:14.123456789-05:00", "419101754123456789")] // full precision
|
|
|
|
#[case("1983-04-13T12:09:14.456789-05:00", "419101754456789000")] // microsec
|
|
|
|
#[case("1983-04-13T12:09:14-05:00", "419101754000000000")] // sec
|
|
|
|
#[case("2052-04-13T12:09:14.123456789-05:00", "2596640954123456789")] // future date > 2038 epoch
|
|
|
|
#[case("1902-04-13T12:09:14.123456789-05:00", "-2137042245876543211")] // past date < 1970
|
|
|
|
fn into_int_datetime(#[case] time_in: &str, #[case] int_out: &str) {
|
2023-07-17 18:43:51 +02:00
|
|
|
let actual = nu!(&format!(
|
|
|
|
r#""{time_in}" | into datetime --format "%+" | into int"#
|
Fix 8244 -- store timestamps with nanosecond resolution (consistently) (#8337)
# Description
Fix for data ambiguity noted in #8244.
Basic change is to use nanosecond resolution for unix timestamps (stored
in type Int). Previously, a timestamp might have seconds, milliseconds
or nanoseconds, but it turned out there were overlaps in data ranges
between different resolutions, so there wasn't always a unique mapping
back to date/time.
Due to higher precision, the *range* of dates that timestamps can map to
is restricted. Unix timestamps with seconds resolution and 64 bit
storage can cover all dates from the Big Bang to eternity. Timestamps
with seconds resolution and 32 bit storage can only represent dates from
1901-12-13 through 2038-01-19. The nanoseconds resolution and 64 bit
storage used with this fix can represent dates from 1677-09-21T00:12:44
to 2262-04-11T23:47:16, something of a compromise.
# User-Facing Changes
_(List of all changes that impact the user experience here. This helps
us keep track of breaking changes.)_
## `<datetime> | into int`
Converts to nanosecond resolution
```rust
〉date now | into int
1678084730502126846
```
This is the number of non-leap nanoseconds after the unix epoch date:
1970-01-01T00:00:00+00:00.
Conversion fails for dates outside the supported range:
```rust
〉1492-10-12 | into int
Error: nu::shell::incorrect_value
× Incorrect value.
╭─[entry #51:1:1]
1 │ 1492-10-12 | into int
· ────┬───
· ╰── DateTime out of timestamp range 1677-09-21T00:12:43 and 2262-04-11T23:47:16
╰────
```
## `<int> | into datetime`
Can no longer fail or produce incorrect results for any 64-bit input:
```rust
〉0 | into datetime
Thu, 01 Jan 1970 00:00:00 +0000 (53 years ago)
〉"7fffffffffffffff" | into int -r 16 | into datetime
Fri, 11 Apr 2262 23:47:16 +0000 (in 239 years)
〉("7fffffffffffffff" | into int -r 16) * -1 | into datetime
Tue, 21 Sep 1677 00:12:43 +0000 (345 years ago)
```
## `<date> | date to-record` and `<date> | date to-table`
Now both have a `nanosecond` field.
```rust
〉"7fffffffffffffff" | into int -r 16 | into datetime | date to-record
╭────────────┬───────────╮
│ year │ 2262 │
│ month │ 4 │
│ day │ 11 │
│ hour │ 23 │
│ minute │ 47 │
│ second │ 16 │
│ nanosecond │ 854775807 │
│ timezone │ +00:00 │
╰────────────┴───────────╯
〉"7fffffffffffffff" | into int -r 16 | into datetime | date to-table
╭───┬──────┬───────┬─────┬──────┬────────┬────────┬────────────┬──────────╮
│ # │ year │ month │ day │ hour │ minute │ second │ nanosecond │ timezone │
├───┼──────┼───────┼─────┼──────┼────────┼────────┼────────────┼──────────┤
│ 0 │ 2262 │ 4 │ 11 │ 23 │ 47 │ 16 │ 854775807 │ +00:00 │
╰───┴──────┴───────┴─────┴──────┴────────┴────────┴────────────┴──────────╯
```
This change was not mandated by the OP problem, but it is nice to be
able to see the nanosecond bits that were present in Nushell `date` type
all along.
# Tests + Formatting
Don't forget to add tests that cover your changes.
Make sure you've run and fixed any issues with these commands:
- `cargo fmt --all -- --check` to check standard code formatting (`cargo
fmt --all` applies these changes)
- `cargo clippy --workspace -- -D warnings -D clippy::unwrap_used -A
clippy::needless_collect` to check that you're using the standard code
style
- `cargo test --workspace` to check that all tests pass
# After Submitting
If your PR had any user-facing changes, update [the
documentation](https://github.com/nushell/nushell.github.io) after the
PR is merged, if necessary. This will help us keep the docs up to date.
2023-03-08 00:02:15 +01:00
|
|
|
));
|
|
|
|
|
|
|
|
assert_eq!(int_out, actual.out);
|
|
|
|
}
|