1854031 - Upgrade segmenter to ICU4X 1.4

Status: RESOLVED FIXED

(Core :: Internationalization, task)

Description

[Ting-Yu Lin [:TYLin] (PDT, UTC-7)]

ICU4X 1.3 is schedule to be released soon, and it fixed bugs in Segmenter. We should upgrade it once it is released.

Comment 1

[Makoto Kato [:m_kato]]

Note that when ICU4X 1.3 with complied_data feature, https://github.com/rust-lang/cargo/issues/10801 occurs. It means that ./mach vendor rust copies unnecessary crates.

Comment 2

[Ting-Yu Lin [:TYLin] (PDT, UTC-7)]

(In reply to Ting-Yu Lin [:TYLin] (UTC-8) from comment #0)

ICU4X 1.3 is schedule to be released soon, and it fixed bugs in Segmenter. We should upgrade it once it is released.

FYI. https://crates.io/crates/icu_segmenter/1.3.2

(In reply to Makoto Kato [:m_kato] from comment #1)

Note that when ICU4X 1.3 with complied_data feature, https://github.com/rust-lang/cargo/issues/10801 occurs. It means that ./mach vendor rust copies unnecessary crates.

Yeah, introducing unnecessary crates into gecko is a pain ... Do we have any workaround in gecko?

Comment 3

[Makoto Kato [:m_kato]]

Also, I file an issue as https://github.com/unicode-org/icu4x/issues/4109. This may be a bug of diplomat_runtime.

Comment 4

[Makoto Kato [:m_kato]]

Additional,

• We don't need dictionary_wl_ext_v1, but when using compiled_data feature, it have to be required.
• License is change to Unicode License V3 (https://www.unicode.org/license.txt). So we have to update python/mozbuild/mozbuild/vendor/vendor_rust.py and get approval for this license.

So we cannot upgrade to 1.3.2. So we have to wait for fixing comment #3 dictionary_wl_ext_v1 issue and at least.

Comment 5

[Makoto Kato [:m_kato]]

https://github.com/unicode-org/icu4x/issues/4111

Comment 6

[Makoto Kato [:m_kato]]

WIP: https://treeherder.mozilla.org/jobs?repo=try&revision=32d0946ae7b35d5cb3619279647f6292b3314d5e

Comment 7

[Makoto Kato [:m_kato]]

New issue: https://github.com/rust-diplomat/diplomat/issues/349

Comment 8

[Makoto Kato [:m_kato]]

Attachment: Bug 1854031 - Part 2. Allow Unicode License V3 to vendor rust for ICU4X 1.4. r=sylvestre

Comment 9

[Makoto Kato [:m_kato]]

Attachment: Bug 1854031 - Part 1. Update update-icu4x.sh script to import icu_capi to local and for datagen change. r=TYLin

Although icu_capi uses weak dependency syntax, cargo vendor doesn't support
it. So this command will copy unnecessary crates. To avoid it, we use modified
version of icu_capi.

Also, icu_capi's C++ headers isn't compatible with clang [*1].

From ICU4X 1.3, there are new icu_*_data crates to custom data file, instead of
icu_testdata.

*1 https://github.com/llvm/llvm-project/issues/70162

Depends on D192900

Comment 10

[Makoto Kato [:m_kato]]

Asking whether Unicode License V3 is allowed (https://phabricator.services.mozilla.com/D193036).

Comment 11

[Makoto Kato [:m_kato]]

Attachment: Bug 1854031 - Part 3. Import icu_capi and icu_segmenter_data crate in tree. r=TYLin

By update-icu4x.sh script.

Depends on D192900

Comment 12

[Makoto Kato [:m_kato]]

Attachment: Bug 1854031 - Part 5. Update Cargo.toml for ICU4X 1.4. r=TYLin!,#supply-chain-reviewers!

Remove unnecessary patches in Cargo.toml

Depends on D193880

Comment 13

[Makoto Kato [:m_kato]]

Attachment: Bug 1854031 - Part 6. Run ./mach vendor rust. r=#supply-chain-reviewers!,TYLin!

Depends on D193881

Comment 14

[Makoto Kato [:m_kato]]

Attachment: Bug 1854031 - Part 7. Gecko changes for ICU4X 1.4. r=TYLin!

Depends on D193882

Comment 15

[Makoto Kato [:m_kato]]

Attachment: WIP: Bug 1854031 - Part 8. Add Unicode License V3. r=sylvestre

Depends on D193883

Comment 16

[Makoto Kato [:m_kato]]

Attachment: Bug 1854031 - Part 4. Update lint file path. r=#linter-reviewers

Depends on D193884

Comment 17

[Makoto Kato [:m_kato]]

Attachment: Bug 1854031 - Part 8. Update ICU4X document. r=TYLin

Depends on D193885

Comment 18

[Makoto Kato [:m_kato]]

Attachment: Bug 1854031 - Part 9. Add Unicode License V3 for ICU4X. r=sylvestre

Since ICU4C 74 is backed out in tree, we don't land bug Bug 1806348 yet.

ICU4X 1.4 uses Unicode License V3, so we have to update license text.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/69f846d2ca22
Part 1. Update update-icu4x.sh script to import icu_capi to local and for datagen change. r=TYLin
https://hg.mozilla.org/integration/autoland/rev/3639285c0b4c
Part 2. Allow Unicode License V3 to vendor rust for ICU4X 1.4. r=glandium
https://hg.mozilla.org/integration/autoland/rev/b3838936ce45
Part 3. Import icu_capi and icu_segmenter_data crate in tree. r=TYLin
https://hg.mozilla.org/integration/autoland/rev/1bf11c6f147b
Part 4. Update lint file path. r=linter-reviewers,sylvestre
https://hg.mozilla.org/integration/autoland/rev/94260c8116da
Part 5. Update Cargo.toml for ICU4X 1.4. r=TYLin
https://hg.mozilla.org/integration/autoland/rev/1478a3bac570
Part 6. Run ./mach vendor rust. r=TYLin,supply-chain-reviewers
https://hg.mozilla.org/integration/autoland/rev/676ce33ea864
Part 7. Gecko changes for ICU4X 1.4. r=TYLin
https://hg.mozilla.org/integration/autoland/rev/980a2ae3ed27
Part 8. Update ICU4X document. r=TYLin
https://hg.mozilla.org/integration/autoland/rev/7b7ac6e2bc52
Part 9. Add Unicode License V3 for ICU4X. r=sylvestre

Comment 21

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/69f846d2ca22
https://hg.mozilla.org/mozilla-central/rev/3639285c0b4c
https://hg.mozilla.org/mozilla-central/rev/b3838936ce45
https://hg.mozilla.org/mozilla-central/rev/1bf11c6f147b
https://hg.mozilla.org/mozilla-central/rev/94260c8116da
https://hg.mozilla.org/mozilla-central/rev/1478a3bac570
https://hg.mozilla.org/mozilla-central/rev/676ce33ea864
https://hg.mozilla.org/mozilla-central/rev/980a2ae3ed27
https://hg.mozilla.org/mozilla-central/rev/7b7ac6e2bc52