Skip to content

Implement ar-file parsing in python #8681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jun 3, 2019
Merged

Implement ar-file parsing in python #8681

merged 3 commits into from
Jun 3, 2019

Conversation

sbc100
Copy link
Collaborator

@sbc100 sbc100 commented May 27, 2019 •
edited
Loading

This takes a different approach to the problem of duplicate files
in ar archives.

This makes the emar wrapper unnecessary solving the problem at ar file
extraction time instead.

Fixes: #8670

@sbc100 sbc100 requested a review from kripken May 27, 2019 17:50
@sbc100 sbc100 force-pushed the arfile branch 3 times, most recently from 8f63e25 to be6a01f Compare May 27, 2019 18:11
kripken
kripken approved these changes May 28, 2019
View reviewed changes
Copy link
Member

@kripken kripken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know the ar file format, but if this passes tests then lgtm!

@sbc100 sbc100 force-pushed the arfile branch 3 times, most recently from b545df0 to 133ad58 Compare May 31, 2019 20:55
sbc100 added 2 commits May 31, 2019 14:24
@sbc100
Implement ar-file parsing in python
be69d61 
This takes a different approach to the problem of duplicate files
in ar archives.

This makes the emar wrapper unnecessary solving the problem at ar file
extraction time instead.
@sbc100
fix determinism
c4c4152
@sbc100 sbc100 merged commit 1d198e4 into incoming Jun 3, 2019
@sbc100 sbc100 deleted the arfile branch June 3, 2019 17:24
@kripken
Copy link
Member

kripken commented Jun 4, 2019

@sbc100 This broke the emscripten-releases Mac bot. That bot was broken on tests before, but this broke the build step.

autoroll: https://chromium-review.googlesource.com/c/emscripten-releases/+/1640373
failing build: https://ci.chromium.org/p/emscripten-releases/builders/ci/mac/b8911649373078606944
build from before for comparison (where the build step passes, but test step later fails): https://ci.chromium.org/p/emscripten-releases/builders/ci/mac/b8911885373953154080

sbc100 added a commit that referenced this pull request Jun 5, 2019
@sbc100
Revert "Implement ar-file parsing in python (#8681)"
bb7119c 
This reverts commit 1d198e4.

Apparently this cause a failure on the mac builder:
https://ci.chromium.org/p/emscripten-releases/builders/ci/mac/b8911649373078606944

```
cache:INFO:  - ok
Traceback (most recent call last):
  File "/b/s/w/ir/k/install/emscripten/emcc.py", line 3391, in <module>
    sys.exit(run(sys.argv))
  File "/b/s/w/ir/k/install/emscripten/emcc.py", line 1894, in run
    final = shared.Building.link(linker_inputs, DEFAULT_FINAL, force_archive_contents=force_archive_contents, just_calculate=just_calculate)
  File "/b/s/w/ir/k/install/emscripten/tools/shared.py", line 1940, in link
    Building.read_link_inputs([x for x in files if not x.startswith('-')])
  File "/b/s/w/ir/k/install/emscripten/tools/shared.py", line 1721, in read_link_inputs
    object_names_in_archives = pool.map(extract_archive_contents, archive_names)
  File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/multiprocessing/pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/multiprocessing/pool.py", line 572, in get
    raise self._value
IOError: [Errno 2] No such file or directory: u'/b/s/w/ir/tmp/t/emscripten_temp_oIE5H7_archive_contents/#1/12'
```
sbc100 added a commit that referenced this pull request Jun 5, 2019
@sbc100
Revert "Implement ar-file parsing in python (#8681)" (#8744)
fc186bd 
This reverts commit 1d198e4.

Apparently this cause a failure on the mac builder:
https://ci.chromium.org/p/emscripten-releases/builders/ci/mac/b8911649373078606944

```
cache:INFO:  - ok
Traceback (most recent call last):
  File "/b/s/w/ir/k/install/emscripten/emcc.py", line 3391, in <module>
    sys.exit(run(sys.argv))
  File "/b/s/w/ir/k/install/emscripten/emcc.py", line 1894, in run
    final = shared.Building.link(linker_inputs, DEFAULT_FINAL, force_archive_contents=force_archive_contents, just_calculate=just_calculate)
  File "/b/s/w/ir/k/install/emscripten/tools/shared.py", line 1940, in link
    Building.read_link_inputs([x for x in files if not x.startswith('-')])
  File "/b/s/w/ir/k/install/emscripten/tools/shared.py", line 1721, in read_link_inputs
    object_names_in_archives = pool.map(extract_archive_contents, archive_names)
  File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/multiprocessing/pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/multiprocessing/pool.py", line 572, in get
    raise self._value
IOError: [Errno 2] No such file or directory: u'/b/s/w/ir/tmp/t/emscripten_temp_oIE5H7_archive_contents/#1/12'
```
belraquib pushed a commit to belraquib/emscripten that referenced this pull request Dec 23, 2020
@sbc100 @belraquib
Implement ar-file parsing in python (emscripten-core#8681)
7a3a818 
This takes a different approach to the problem of duplicate files
in ar archives.

This makes the emar wrapper unnecessary solving the problem at ar file
extraction time instead.
belraquib pushed a commit to belraquib/emscripten that referenced this pull request Dec 23, 2020
@sbc100 @belraquib
Revert "Implement ar-file parsing in python (emscripten-core#8681)" (e…
045f60f 
…mscripten-core#8744)

This reverts commit 1d198e4.

Apparently this cause a failure on the mac builder:
https://ci.chromium.org/p/emscripten-releases/builders/ci/mac/b8911649373078606944

```
cache:INFO:  - ok
Traceback (most recent call last):
  File "/b/s/w/ir/k/install/emscripten/emcc.py", line 3391, in <module>
    sys.exit(run(sys.argv))
  File "/b/s/w/ir/k/install/emscripten/emcc.py", line 1894, in run
    final = shared.Building.link(linker_inputs, DEFAULT_FINAL, force_archive_contents=force_archive_contents, just_calculate=just_calculate)
  File "/b/s/w/ir/k/install/emscripten/tools/shared.py", line 1940, in link
    Building.read_link_inputs([x for x in files if not x.startswith('-')])
  File "/b/s/w/ir/k/install/emscripten/tools/shared.py", line 1721, in read_link_inputs
    object_names_in_archives = pool.map(extract_archive_contents, archive_names)
  File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/multiprocessing/pool.py", line 253, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/b/s/w/ir/cipd_bin_packages/lib/python2.7/multiprocessing/pool.py", line 572, in get
    raise self._value
IOError: [Errno 2] No such file or directory: u'/b/s/w/ir/tmp/t/emscripten_temp_oIE5H7_archive_contents/#1/12'
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kripken kripken kripken approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

emar adds object files twice
2 participants
@sbc100 @kripken