* [Tarantool-patches] [PATCH luajit 1/4][v2] ci: add a workflow for testing with AVX512
2024-04-11 13:22 [Tarantool-patches] [PATCH luajit 0/4][v2] Mach-O generation fixes Sergey Bronnikov via Tarantool-patches
@ 2024-04-11 13:22 ` Sergey Bronnikov via Tarantool-patches
2024-04-12 10:27 ` Sergey Kaplun via Tarantool-patches
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 2/4][v2] test: introduce a helper read_file Sergey Bronnikov via Tarantool-patches
` (2 subsequent siblings)
3 siblings, 1 reply; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2024-04-11 13:22 UTC (permalink / raw)
To: tarantool-patches, Sergey Kaplun, Maxim Kokryashkin
From: Sergey Bronnikov <sergeyb@tarantool.org>
This commit adds a workflow for building and testing with enabled
AVX512.
Needed for tarantool/tarantool#9595
Related to tarantool/tarantool#6787
---
.github/actions/setup-linux/action.yml | 12 +++++
.github/workflows/avx512-build-testing.yml | 54 ++++++++++++++++++++++
test/LuaJIT-tests/lib/ffi/bit64.lua | 2 +-
3 files changed, 67 insertions(+), 1 deletion(-)
create mode 100644 .github/workflows/avx512-build-testing.yml
diff --git a/.github/actions/setup-linux/action.yml b/.github/actions/setup-linux/action.yml
index f0171b83..19bdcfa2 100644
--- a/.github/actions/setup-linux/action.yml
+++ b/.github/actions/setup-linux/action.yml
@@ -17,3 +17,15 @@ runs:
apt -y update
apt -y install cmake gcc make ninja-build perl
shell: bash
+ - name: Detect a presence of AVX512
+ id: avx512_script
+ run: |
+ # Set avx512_support environment variable to 'true' when AVX512
+ # is supported and 'false' otherwise.
+ #
+ # Normally `grep` has the exit status is 0 if a line is
+ # selected, 1 if no lines were selected, and 2 if an error
+ # occurred.
+ avx512_support=$(grep avx /proc/cpuinfo 2>&1 > /dev/null; [[ $? == 0 ]] && echo 0 || echo 1)
+ echo "avx512_support=$avx512_support" >> $GITHUB_OUTPUT
+ shell: bash
diff --git a/.github/workflows/avx512-build-testing.yml b/.github/workflows/avx512-build-testing.yml
new file mode 100644
index 00000000..d70fa661
--- /dev/null
+++ b/.github/workflows/avx512-build-testing.yml
@@ -0,0 +1,54 @@
+name: "AVX512 build testing"
+
+on:
+ push:
+ branches-ignore:
+ - '**-notest'
+ - 'upstream-**'
+ tags-ignore:
+ - '**'
+
+concurrency:
+ # An update of a developer branch cancels the previously
+ # scheduled workflow run for this branch. However, the default
+ # branch, and long-term branch (tarantool/release/2.11,
+ # tarantool/release/2.10, etc) workflow runs are never canceled.
+ #
+ # We use a trick here: define the concurrency group as 'workflow
+ # run ID' + # 'workflow run attempt' because it is a unique
+ # combination for any run. So it effectively discards grouping.
+ #
+ # XXX: we cannot use `github.sha` as a unique identifier because
+ # pushing a tag may cancel a run that works on a branch push
+ # event.
+ group: ${{ startsWith(github.ref, 'refs/heads/tarantool/')
+ && format('{0}-{1}', github.run_id, github.run_attempt)
+ || format('{0}-{1}', github.workflow, github.ref) }}
+ cancel-in-progress: true
+
+jobs:
+ test-avx512:
+ strategy:
+ fail-fast: false
+ runs-on: [self-hosted, regular, x86_64, Linux]
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ submodules: recursive
+ - name: setup Linux
+ uses: ./.github/actions/setup-linux
+ - name: configure
+ if: needs.avx512_script.outputs.avx512_support == 0
+ run: >
+ cmake -S . -B ${{ env.BUILDDIR }}
+ -G Ninja
+ -DCMAKE_BUILD_TYPE=RelWithDebInfo
+ -DCMAKE_C_FLAGS=-march=skylake-avx512
+ -DCMAKE_C_COMPILER=gcc
+ - name: build
+ if: needs.avx512_script.outputs.avx512_support == 0
+ run: cmake --build ${{ env.BUILDDIR }} --parallel
+ - name: run regression tests
+ if: needs.avx512_script.outputs.avx512_support == 0
+ run: cmake --build ${{ env.BUILDDIR }} --parallel --target LuaJIT-test
diff --git a/test/LuaJIT-tests/lib/ffi/bit64.lua b/test/LuaJIT-tests/lib/ffi/bit64.lua
index d1b47bef..cf3a96eb 100644
--- a/test/LuaJIT-tests/lib/ffi/bit64.lua
+++ b/test/LuaJIT-tests/lib/ffi/bit64.lua
@@ -41,7 +41,7 @@ do --- tobit/band assorted C types
end
end
-do --- tobit/band negative unsigned enum
+do --- tobit/band negative unsigned enum -avx512
local x = ffi.new("uenum_t", -10)
local y = tobit(x)
local z = band(x)
--
2.34.1
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Tarantool-patches] [PATCH luajit 1/4][v2] ci: add a workflow for testing with AVX512
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 1/4][v2] ci: add a workflow for testing with AVX512 Sergey Bronnikov via Tarantool-patches
@ 2024-04-12 10:27 ` Sergey Kaplun via Tarantool-patches
2024-04-16 11:53 ` Sergey Bronnikov via Tarantool-patches
0 siblings, 1 reply; 14+ messages in thread
From: Sergey Kaplun via Tarantool-patches @ 2024-04-12 10:27 UTC (permalink / raw)
To: Sergey Bronnikov; +Cc: tarantool-patches
Hi, Sergey!
Thanks for the patch!
Please consider my comments below.
On 11.04.24, Sergey Bronnikov wrote:
> From: Sergey Bronnikov <sergeyb@tarantool.org>
>
> This commit adds a workflow for building and testing with enabled
> AVX512.
Typo: s/with enabled AVX512/with AVX512 enabled/
>
> Needed for tarantool/tarantool#9595
> Related to tarantool/tarantool#6787
Typo: s/Related/Relates/
| git log | grep Relates | wc -l
| 29
| git log | grep Related | wc -l
| 7
> ---
> .github/actions/setup-linux/action.yml | 12 +++++
> .github/workflows/avx512-build-testing.yml | 54 ++++++++++++++++++++++
> test/LuaJIT-tests/lib/ffi/bit64.lua | 2 +-
> 3 files changed, 67 insertions(+), 1 deletion(-)
> create mode 100644 .github/workflows/avx512-build-testing.yml
>
> diff --git a/.github/actions/setup-linux/action.yml b/.github/actions/setup-linux/action.yml
> index f0171b83..19bdcfa2 100644
> --- a/.github/actions/setup-linux/action.yml
> +++ b/.github/actions/setup-linux/action.yml
> @@ -17,3 +17,15 @@ runs:
> apt -y update
> apt -y install cmake gcc make ninja-build perl
> shell: bash
> + - name: Detect a presence of AVX512
I suppose it shouldn't be a part of setup-linux action, but just a one
step in testing-avx512.yml, see [1] for details of steps content usage.
Is it possible or did I miss some GH-action API trick here?
But anyway it should be done not in the same action.
> + id: avx512_script
> + run: |
> + # Set avx512_support environment variable to 'true' when AVX512
Typo: s/avx512_support/the avx512_support/
> + # is supported and 'false' otherwise.
The comment is a little bit misleading since the resulting values are
"0" or "1". Changing it to 'true' or 'false' looks like a good idea and
simplifies reading later.
> + #
> + # Normally `grep` has the exit status is 0 if a line is
Typo: s/Normally/Normally,/
Also, "has the exit status is" has two verbs, so I suggest to
reformulate this like the following:
| Normally, the exit status of `grep` is 0 if a line is selected, 1 if
| no lines were selected, and 2 if an error occurred.
> + # selected, 1 if no lines were selected, and 2 if an error
> + # occurred.
> + avx512_support=$(grep avx /proc/cpuinfo 2>&1 > /dev/null; [[ $? == 0 ]] && echo 0 || echo 1)
Why not just echo the "$?"?
On my laptop without avx512 support this command returns true.
| $ grep avx /proc/cpuinfo 2>&1 > /dev/null; [[ $? == 0 ]] && echo 0 || echo 1
| 0
I have no avx512 support, but avx|avx2 support. Hence, there is matching
content:
| $ grep -o -P '.{4}avx.{4}' /proc/cpuinfo
| ave avx f16
| mi1 avx2 sm
| ave avx f16
| mi1 avx2 sm
| ave avx f16
| mi1 avx2 sm
| ave avx f16
| mi1 avx2 sm
I suppose we should check for the "AVX512F" CPUID feature flag: its
supports allows to emit vcvttsd2usi [2] instruction that causes troubles.
> + echo "avx512_support=$avx512_support" >> $GITHUB_OUTPUT
Minor: It will be nice to echo some message about the value of this
variable. Hence, we can check this from the GitHub logs.
Side note: Do I understand correctly that GH-actions output is always a
string that evaluates to `true` regardless of its content?
> + shell: bash
> diff --git a/.github/workflows/avx512-build-testing.yml b/.github/workflows/avx512-build-testing.yml
> new file mode 100644
> index 00000000..d70fa661
> --- /dev/null
> +++ b/.github/workflows/avx512-build-testing.yml
<snipped>
> +
> +jobs:
> + test-avx512:
Minor: I suggest adding a name for the job to be consistent with other
jobs. Something like "LuaJIT avx512""
> + strategy:
> + fail-fast: false
> + runs-on: [self-hosted, regular, x86_64, Linux]
> + steps:
> + - uses: actions/checkout@v4
> + with:
> + fetch-depth: 0
> + submodules: recursive
> + - name: setup Linux
> + uses: ./.github/actions/setup-linux
> + - name: configure
> + if: needs.avx512_script.outputs.avx512_support == 0
> + run: >
> + cmake -S . -B ${{ env.BUILDDIR }}
> + -G Ninja
> + -DCMAKE_BUILD_TYPE=RelWithDebInfo
Please add a comment. Why do we check the non-debug build here?
If there is no reason for it, maybe it is better to check both: the
debug and release builds within a simple matrix.
> + -DCMAKE_C_FLAGS=-march=skylake-avx512
> + -DCMAKE_C_COMPILER=gcc
Do we need to specify compiler manually?
> + - name: build
> + if: needs.avx512_script.outputs.avx512_support == 0
> + run: cmake --build ${{ env.BUILDDIR }} --parallel
> + - name: run regression tests
> + if: needs.avx512_script.outputs.avx512_support == 0
> + run: cmake --build ${{ env.BUILDDIR }} --parallel --target LuaJIT-test
> diff --git a/test/LuaJIT-tests/lib/ffi/bit64.lua b/test/LuaJIT-tests/lib/ffi/bit64.lua
> index d1b47bef..cf3a96eb 100644
> --- a/test/LuaJIT-tests/lib/ffi/bit64.lua
> +++ b/test/LuaJIT-tests/lib/ffi/bit64.lua
> @@ -41,7 +41,7 @@ do --- tobit/band assorted C types
> end
> end
>
> -do --- tobit/band negative unsigned enum
> +do --- tobit/band negative unsigned enum -avx512
Should we also to set such variable in the CMakeLists.txt when we met
the corresponding condition?
> local x = ffi.new("uenum_t", -10)
> local y = tobit(x)
> local z = band(x)
> --
> 2.34.1
[1]: https://docs.github.com/en/actions/learn-github-actions/contexts#example-usage-of-the-steps-context
[2]: https://www.laruence.com/x86/VCVTTSD2USI.html
--
Best regards,
Sergey Kaplun
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Tarantool-patches] [PATCH luajit 1/4][v2] ci: add a workflow for testing with AVX512
2024-04-12 10:27 ` Sergey Kaplun via Tarantool-patches
@ 2024-04-16 11:53 ` Sergey Bronnikov via Tarantool-patches
2024-04-18 8:24 ` Sergey Kaplun via Tarantool-patches
0 siblings, 1 reply; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2024-04-16 11:53 UTC (permalink / raw)
To: Sergey Kaplun, Sergey Bronnikov; +Cc: tarantool-patches
[-- Attachment #1: Type: text/plain, Size: 6122 bytes --]
Hi, Sergey
On 12.04.2024 13:27, Sergey Kaplun wrote:
> Hi, Sergey!
> Thanks for the patch!
> Please consider my comments below.
>
> On 11.04.24, Sergey Bronnikov wrote:
>> From: Sergey Bronnikov<sergeyb@tarantool.org>
>>
>> This commit adds a workflow for building and testing with enabled
>> AVX512.
> Typo: s/with enabled AVX512/with AVX512 enabled/
Fixed.
>
>> Needed for tarantool/tarantool#9595
>> Related to tarantool/tarantool#6787
> Typo: s/Related/Relates/
Fixed.
>
> | git log | grep Relates | wc -l
> | 29
> | git log | grep Related | wc -l
> | 7
>> ---
>> .github/actions/setup-linux/action.yml | 12 +++++
>> .github/workflows/avx512-build-testing.yml | 54 ++++++++++++++++++++++
>> test/LuaJIT-tests/lib/ffi/bit64.lua | 2 +-
>> 3 files changed, 67 insertions(+), 1 deletion(-)
>> create mode 100644 .github/workflows/avx512-build-testing.yml
>>
>> diff --git a/.github/actions/setup-linux/action.yml b/.github/actions/setup-linux/action.yml
>> index f0171b83..19bdcfa2 100644
>> --- a/.github/actions/setup-linux/action.yml
>> +++ b/.github/actions/setup-linux/action.yml
>> @@ -17,3 +17,15 @@ runs:
>> apt -y update
>> apt -y install cmake gcc make ninja-build perl
>> shell: bash
>> + - name: Detect a presence of AVX512
> I suppose it shouldn't be a part of setup-linux action,
It was requested by Max.
> but just a one
> step in testing-avx512.yml, see [1] for details of steps content usage.
>
> Is it possible or did I miss some GH-action API trick here?
> But anyway it should be done not in the same action.
>
>> + id: avx512_script
>> + run: |
>> + # Set avx512_support environment variable to 'true' when AVX512
> Typo: s/avx512_support/the avx512_support/
Fixed.
>> + # is supported and 'false' otherwise.
> The comment is a little bit misleading since the resulting values are
> "0" or "1". Changing it to 'true' or 'false' looks like a good idea and
> simplifies reading later.
Fixed.
>
>> + #
>> + # Normally `grep` has the exit status is 0 if a line is
> Typo: s/Normally/Normally,/
> Also, "has the exit status is" has two verbs, so I suggest to
> reformulate this like the following:
>
> | Normally, the exit status of `grep` is 0 if a line is selected, 1 if
> | no lines were selected, and 2 if an error occurred.
>
>> + # selected, 1 if no lines were selected, and 2 if an error
>> + # occurred.
>> + avx512_support=$(grep avx /proc/cpuinfo 2>&1 > /dev/null; [[ $? == 0 ]] && echo 0 || echo 1)
> Why not just echo the "$?"?
because comment above. `grep` has three possible values.
>
> On my laptop without avx512 support this command returns true.
> | $ grep avx /proc/cpuinfo 2>&1 > /dev/null; [[ $? == 0 ]] && echo 0 || echo 1
> | 0
>
> I have no avx512 support, but avx|avx2 support. Hence, there is matching
> content:
>
> | $ grep -o -P '.{4}avx.{4}' /proc/cpuinfo
> | ave avx f16
> | mi1 avx2 sm
> | ave avx f16
> | mi1 avx2 sm
> | ave avx f16
> | mi1 avx2 sm
> | ave avx f16
> | mi1 avx2 sm
>
> I suppose we should check for the "AVX512F" CPUID feature flag: its
> supports allows to emit vcvttsd2usi [2] instruction that causes troubles.
Changed to AVX512F.
>
>> + echo "avx512_support=$avx512_support" >> $GITHUB_OUTPUT
> Minor: It will be nice to echo some message about the value of this
> variable. Hence, we can check this from the GitHub logs.
What for? Job will be skipped if there is no AVX512 support.
What is the value of the message?
>
> Side note: Do I understand correctly that GH-actions output is always a
> string that evaluates to `true` regardless of its content?
0/1
>
>> + shell: bash
>> diff --git a/.github/workflows/avx512-build-testing.yml b/.github/workflows/avx512-build-testing.yml
>> new file mode 100644
>> index 00000000..d70fa661
>> --- /dev/null
>> +++ b/.github/workflows/avx512-build-testing.yml
> <snipped>
>
>> +
>> +jobs:
>> + test-avx512:
> Minor: I suggest adding a name for the job to be consistent with other
> jobs. Something like "LuaJIT avx512""
Added.
>
>> + strategy:
>> + fail-fast: false
>> + runs-on: [self-hosted, regular, x86_64, Linux]
>> + steps:
>> + - uses: actions/checkout@v4
>> + with:
>> + fetch-depth: 0
>> + submodules: recursive
>> + - name: setup Linux
>> + uses: ./.github/actions/setup-linux
>> + - name: configure
>> + if: needs.avx512_script.outputs.avx512_support == 0
>> + run: >
>> + cmake -S . -B ${{ env.BUILDDIR }}
>> + -G Ninja
>> + -DCMAKE_BUILD_TYPE=RelWithDebInfo
> Please add a comment. Why do we check the non-debug build here?
> If there is no reason for it, maybe it is better to check both: the
> debug and release builds within a simple matrix.
There is no reason for this.
>
>> + -DCMAKE_C_FLAGS=-march=skylake-avx512
>> + -DCMAKE_C_COMPILER=gcc
> Do we need to specify compiler manually?
Yes, CFLAG is gcc-specific.
>
>> + - name: build
>> + if: needs.avx512_script.outputs.avx512_support == 0
>> + run: cmake --build ${{ env.BUILDDIR }} --parallel
>> + - name: run regression tests
>> + if: needs.avx512_script.outputs.avx512_support == 0
>> + run: cmake --build ${{ env.BUILDDIR }} --parallel --target LuaJIT-test
>> diff --git a/test/LuaJIT-tests/lib/ffi/bit64.lua b/test/LuaJIT-tests/lib/ffi/bit64.lua
>> index d1b47bef..cf3a96eb 100644
>> --- a/test/LuaJIT-tests/lib/ffi/bit64.lua
>> +++ b/test/LuaJIT-tests/lib/ffi/bit64.lua
>> @@ -41,7 +41,7 @@ do --- tobit/band assorted C types
>> end
>> end
>>
>> -do --- tobit/band negative unsigned enum
>> +do --- tobit/band negative unsigned enum -avx512
> Should we also to set such variable in the CMakeLists.txt when we met
> the corresponding condition?
Yes, double check.
>
>> local x = ffi.new("uenum_t", -10)
>> local y = tobit(x)
>> local z = band(x)
>> --
>> 2.34.1
> [1]:https://docs.github.com/en/actions/learn-github-actions/contexts#example-usage-of-the-steps-context
> [2]:https://www.laruence.com/x86/VCVTTSD2USI.html
>
[-- Attachment #2: Type: text/html, Size: 10676 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Tarantool-patches] [PATCH luajit 1/4][v2] ci: add a workflow for testing with AVX512
2024-04-16 11:53 ` Sergey Bronnikov via Tarantool-patches
@ 2024-04-18 8:24 ` Sergey Kaplun via Tarantool-patches
2024-05-05 12:29 ` Maxim Kokryashkin via Tarantool-patches
0 siblings, 1 reply; 14+ messages in thread
From: Sergey Kaplun via Tarantool-patches @ 2024-04-18 8:24 UTC (permalink / raw)
To: Sergey Bronnikov; +Cc: Sergey Bronnikov, tarantool-patches
Hi, Sergey
On 16.04.24, Sergey Bronnikov wrote:
> Hi, Sergey
>
> On 12.04.2024 13:27, Sergey Kaplun wrote:
<snipped>
> >> diff --git a/.github/actions/setup-linux/action.yml b/.github/actions/setup-linux/action.yml
> >> index f0171b83..19bdcfa2 100644
> >> --- a/.github/actions/setup-linux/action.yml
> >> +++ b/.github/actions/setup-linux/action.yml
> >> @@ -17,3 +17,15 @@ runs:
> >> apt -y update
> >> apt -y install cmake gcc make ninja-build perl
> >> shell: bash
> >> + - name: Detect a presence of AVX512
> > I suppose it shouldn't be a part of setup-linux action,
>
> It was requested by Max.
And what is the rationale for it?
Why don't use this detection during steps in our workflow?
Hence, we avoid this unnecesary step for other jobs.
>
>
> > but just a one
> > step in testing-avx512.yml, see [1] for details of steps content usage.
> >
> > Is it possible or did I miss some GH-action API trick here?
> > But anyway it should be done not in the same action.
> >
<snipped>
> >
> >> + #
> >> + # Normally `grep` has the exit status is 0 if a line is
> > Typo: s/Normally/Normally,/
> > Also, "has the exit status is" has two verbs, so I suggest to
> > reformulate this like the following:
> >
> > | Normally, the exit status of `grep` is 0 if a line is selected, 1 if
> > | no lines were selected, and 2 if an error occurred.
Ping.
> >
> >> + # selected, 1 if no lines were selected, and 2 if an error
> >> + # occurred.
> >> + avx512_support=$(grep avx /proc/cpuinfo 2>&1 > /dev/null; [[ $? == 0 ]] && echo 0 || echo 1)
> > Why not just echo the "$?"?
> because comment above. `grep` has three possible values.
Yes, but why do we need such tricks if we are still interested in the 0
value only? Also, with this and the logging of the `avx512_support`, we
may see the reason why the workflow was skipped: an incorrect `grep`
command (for some reason) or a not-founded CPU feature flag.
Hence, if the command fails with exit code 2, that means that our CI
settings are incorrect and need to be adjusted.
> >
<snipped>
> >
> >> + echo "avx512_support=$avx512_support" >> $GITHUB_OUTPUT
> > Minor: It will be nice to echo some message about the value of this
> > variable. Hence, we can check this from the GitHub logs.
>
> What for? Job will be skipped if there is no AVX512 support.
>
> What is the value of the message?
See the rationale above.
>
>
> >
<snipped>
> >> + if: needs.avx512_script.outputs.avx512_support == 0
> >> + run: >
> >> + cmake -S . -B ${{ env.BUILDDIR }}
> >> + -G Ninja
> >> + -DCMAKE_BUILD_TYPE=RelWithDebInfo
> > Please add a comment. Why do we check the non-debug build here?
> > If there is no reason for it, maybe it is better to check both: the
> > debug and release builds within a simple matrix.
> There is no reason for this.
So, let's check the Debug build only. And with enabled assertions.
They are useful for testing.
> >
> >> + -DCMAKE_C_FLAGS=-march=skylake-avx512
> >> + -DCMAKE_C_COMPILER=gcc
> > Do we need to specify compiler manually?
> Yes, CFLAG is gcc-specific.
Emm, is it?
| $ clang -march=skylake-avx512 /tmp/t.c -o /tmp/t.exe
| $ clang --version
| clang version 17.0.6
| Target: x86_64-pc-linux-gnu
| Thread model: posix
| InstalledDir: /usr/lib/llvm/17/bin
| Configuration file: /etc/clang/x86_64-pc-linux-gnu-clang.cfg
Maybe it is unsupported for clang version we used in CI?
> >
> >> + - name: build
> >> + if: needs.avx512_script.outputs.avx512_support == 0
> >> + run: cmake --build ${{ env.BUILDDIR }} --parallel
> >> + - name: run regression tests
> >> + if: needs.avx512_script.outputs.avx512_support == 0
> >> + run: cmake --build ${{ env.BUILDDIR }} --parallel --target LuaJIT-test
> >> diff --git a/test/LuaJIT-tests/lib/ffi/bit64.lua b/test/LuaJIT-tests/lib/ffi/bit64.lua
> >> index d1b47bef..cf3a96eb 100644
> >> --- a/test/LuaJIT-tests/lib/ffi/bit64.lua
> >> +++ b/test/LuaJIT-tests/lib/ffi/bit64.lua
> >> @@ -41,7 +41,7 @@ do --- tobit/band assorted C types
> >> end
> >> end
> >>
> >> -do --- tobit/band negative unsigned enum
> >> +do --- tobit/band negative unsigned enum -avx512
> > Should we also to set such variable in the CMakeLists.txt when we met
> > the corresponding condition?
> Yes, double check.
I meant that part of the third commit should be done within this patch,
as you've already fixed.
> >
> >> local x = ffi.new("uenum_t", -10)
> >> local y = tobit(x)
> >> local z = band(x)
> >> --
> >> 2.34.1
> > [1]:https://docs.github.com/en/actions/learn-github-actions/contexts#example-usage-of-the-steps-context
> > [2]:https://www.laruence.com/x86/VCVTTSD2USI.html
> >
--
Best regards,
Sergey Kaplun
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Tarantool-patches] [PATCH luajit 1/4][v2] ci: add a workflow for testing with AVX512
2024-04-18 8:24 ` Sergey Kaplun via Tarantool-patches
@ 2024-05-05 12:29 ` Maxim Kokryashkin via Tarantool-patches
0 siblings, 0 replies; 14+ messages in thread
From: Maxim Kokryashkin via Tarantool-patches @ 2024-05-05 12:29 UTC (permalink / raw)
To: Sergey Kaplun; +Cc: Sergey Bronnikov, tarantool-patches
Hi!
On Thu, Apr 18, 2024 at 11:24:42AM UTC, Sergey Kaplun wrote:
> Hi, Sergey
>
> On 16.04.24, Sergey Bronnikov wrote:
> > Hi, Sergey
> >
> > On 12.04.2024 13:27, Sergey Kaplun wrote:
>
> <snipped>
>
> > >> diff --git a/.github/actions/setup-linux/action.yml b/.github/actions/setup-linux/action.yml
> > >> index f0171b83..19bdcfa2 100644
> > >> --- a/.github/actions/setup-linux/action.yml
> > >> +++ b/.github/actions/setup-linux/action.yml
> > >> @@ -17,3 +17,15 @@ runs:
> > >> apt -y update
> > >> apt -y install cmake gcc make ninja-build perl
> > >> shell: bash
> > >> + - name: Detect a presence of AVX512
> > > I suppose it shouldn't be a part of setup-linux action,
> >
> > It was requested by Max.
>
> And what is the rationale for it?
> Why don't use this detection during steps in our workflow?
>
> Hence, we avoid this unnecesary step for other jobs.
I've suggested to move that to a separate action, not to the
setup-linux.
<snipped>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Tarantool-patches] [PATCH luajit 2/4][v2] test: introduce a helper read_file
2024-04-11 13:22 [Tarantool-patches] [PATCH luajit 0/4][v2] Mach-O generation fixes Sergey Bronnikov via Tarantool-patches
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 1/4][v2] ci: add a workflow for testing with AVX512 Sergey Bronnikov via Tarantool-patches
@ 2024-04-11 13:22 ` Sergey Bronnikov via Tarantool-patches
2024-04-12 10:47 ` Sergey Kaplun via Tarantool-patches
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 3/4][v2] OSX/iOS/ARM64: Fix generation of Mach-O object files Sergey Bronnikov via Tarantool-patches
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 4/4][v2] OSX/iOS/ARM64: Fix bytecode embedding in Mach-O object file Sergey Bronnikov via Tarantool-patches
3 siblings, 1 reply; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2024-04-11 13:22 UTC (permalink / raw)
To: tarantool-patches, Sergey Kaplun, Maxim Kokryashkin
From: Sergey Bronnikov <sergeyb@tarantool.org>
The test `lj-366-strtab-correct-size.test.lua` has a test helper
`read_file` that reads a file's content and returns it.
This helper will be useful for a test upcoming in the next commit,
so it is moved to test tools.
Needed for tarantool/tarantool#9595
---
.../lj-366-strtab-correct-size.test.lua | 13 +++----------
test/tarantool-tests/utils/tools.lua | 8 ++++++++
2 files changed, 11 insertions(+), 10 deletions(-)
diff --git a/test/tarantool-tests/lj-366-strtab-correct-size.test.lua b/test/tarantool-tests/lj-366-strtab-correct-size.test.lua
index 8a97a441..580fce09 100644
--- a/test/tarantool-tests/lj-366-strtab-correct-size.test.lua
+++ b/test/tarantool-tests/lj-366-strtab-correct-size.test.lua
@@ -9,6 +9,7 @@ local test = tap.test('lj-366-strtab-correct-size'):skipcond({
})
local ffi = require 'ffi'
+local utils = require('utils')
-- Command below exports bytecode as an object file in ELF format:
-- $ luajit -b -n 'lango_team' -e 'print()' xxx.obj
@@ -130,7 +131,7 @@ local is64 = is64_arch[jit.arch]
local function create_obj_file(name)
local elf_filename = os.tmpname() .. '.obj'
local lua_path = os.getenv('LUA_PATH')
- local lua_bin = require('utils').exec.luacmd(arg):match('%S+')
+ local lua_bin = utils.exec.luacmd(arg):match('%S+')
local cmd_fmt = 'LUA_PATH="%s" %s -b -n "%s" -e "print()" %s'
local cmd = (cmd_fmt):format(lua_path, lua_bin, name, elf_filename)
local ret = os.execute(cmd)
@@ -138,14 +139,6 @@ local function create_obj_file(name)
return elf_filename
end
--- Reads a file located in a specified path and returns its content.
-local function read_file(path)
- local file = assert(io.open(path), 'cannot open an object file')
- local content = file:read('*a')
- file:close()
- return content
-end
-
-- Parses a buffer in an ELF format and returns an offset and a size of strtab
-- and symtab sections.
local function read_elf(elf_content)
@@ -172,7 +165,7 @@ end
test:plan(3)
local elf_filename = create_obj_file(MODULE_NAME)
-local elf_content = read_file(elf_filename)
+local elf_content = require('utils').tools.read_file(elf_filename)
assert(#elf_content ~= 0, 'cannot read an object file')
local strtab, symtab = read_elf(elf_content)
diff --git a/test/tarantool-tests/utils/tools.lua b/test/tarantool-tests/utils/tools.lua
index f35c6922..a2556e32 100644
--- a/test/tarantool-tests/utils/tools.lua
+++ b/test/tarantool-tests/utils/tools.lua
@@ -12,4 +12,12 @@ function M.profilename(name)
return (arg[0]:gsub('^(.+)/([^/]+)%.test%.lua$', replacepattern))
end
+-- Reads a file located at a specified path and returns its content.
+function M.read_file(path)
+ local file = assert(io.open(path), 'cannot open an object file')
+ local content = file:read('*a')
+ file:close()
+ return content
+end
+
return M
--
2.34.1
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Tarantool-patches] [PATCH luajit 2/4][v2] test: introduce a helper read_file
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 2/4][v2] test: introduce a helper read_file Sergey Bronnikov via Tarantool-patches
@ 2024-04-12 10:47 ` Sergey Kaplun via Tarantool-patches
2024-04-16 12:02 ` Sergey Bronnikov via Tarantool-patches
0 siblings, 1 reply; 14+ messages in thread
From: Sergey Kaplun via Tarantool-patches @ 2024-04-12 10:47 UTC (permalink / raw)
To: Sergey Bronnikov; +Cc: tarantool-patches
Hi, Sergey!
Thanks for the fixes!
LGTM, after fixing two nits below.
On 11.04.24, Sergey Bronnikov wrote:
> From: Sergey Bronnikov <sergeyb@tarantool.org>
>
<snipped>
> diff --git a/test/tarantool-tests/lj-366-strtab-correct-size.test.lua b/test/tarantool-tests/lj-366-strtab-correct-size.test.lua
> index 8a97a441..580fce09 100644
> --- a/test/tarantool-tests/lj-366-strtab-correct-size.test.lua
> +++ b/test/tarantool-tests/lj-366-strtab-correct-size.test.lua
> @@ -9,6 +9,7 @@ local test = tap.test('lj-366-strtab-correct-size'):skipcond({
> })
>
> local ffi = require 'ffi'
> +local utils = require('utils')
<snipped>
> -- Parses a buffer in an ELF format and returns an offset and a size of strtab
> -- and symtab sections.
> local function read_elf(elf_content)
> @@ -172,7 +165,7 @@ end
> test:plan(3)
>
> local elf_filename = create_obj_file(MODULE_NAME)
> -local elf_content = read_file(elf_filename)
> +local elf_content = require('utils').tools.read_file(elf_filename)
Nit: s/require('utils')/utils/
Rationale: `utils` are already required above.
> assert(#elf_content ~= 0, 'cannot read an object file')
>
> local strtab, symtab = read_elf(elf_content)
> diff --git a/test/tarantool-tests/utils/tools.lua b/test/tarantool-tests/utils/tools.lua
> index f35c6922..a2556e32 100644
> --- a/test/tarantool-tests/utils/tools.lua
> +++ b/test/tarantool-tests/utils/tools.lua
> @@ -12,4 +12,12 @@ function M.profilename(name)
> return (arg[0]:gsub('^(.+)/([^/]+)%.test%.lua$', replacepattern))
> end
>
> +-- Reads a file located at a specified path and returns its content.
Nit: Comment line width is more than 66 symbols.
> +function M.read_file(path)
> + local file = assert(io.open(path), 'cannot open an object file')
> + local content = file:read('*a')
> + file:close()
> + return content
> +end
> +
> return M
> --
> 2.34.1
>
--
Best regards,
Sergey Kaplun
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Tarantool-patches] [PATCH luajit 2/4][v2] test: introduce a helper read_file
2024-04-12 10:47 ` Sergey Kaplun via Tarantool-patches
@ 2024-04-16 12:02 ` Sergey Bronnikov via Tarantool-patches
0 siblings, 0 replies; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2024-04-16 12:02 UTC (permalink / raw)
To: Sergey Kaplun, Sergey Bronnikov; +Cc: tarantool-patches
[-- Attachment #1: Type: text/plain, Size: 2013 bytes --]
Hi, Sergey
On 12.04.2024 13:47, Sergey Kaplun wrote:
> Hi, Sergey!
> Thanks for the fixes!
> LGTM, after fixing two nits below.
>
> On 11.04.24, Sergey Bronnikov wrote:
>> From: Sergey Bronnikov<sergeyb@tarantool.org>
>>
> <snipped>
>
>> diff --git a/test/tarantool-tests/lj-366-strtab-correct-size.test.lua b/test/tarantool-tests/lj-366-strtab-correct-size.test.lua
>> index 8a97a441..580fce09 100644
>> --- a/test/tarantool-tests/lj-366-strtab-correct-size.test.lua
>> +++ b/test/tarantool-tests/lj-366-strtab-correct-size.test.lua
>> @@ -9,6 +9,7 @@ local test = tap.test('lj-366-strtab-correct-size'):skipcond({
>> })
>>
>> local ffi = require 'ffi'
>> +local utils = require('utils')
> <snipped>
>
>> -- Parses a buffer in an ELF format and returns an offset and a size of strtab
>> -- and symtab sections.
>> local function read_elf(elf_content)
>> @@ -172,7 +165,7 @@ end
>> test:plan(3)
>>
>> local elf_filename = create_obj_file(MODULE_NAME)
>> -local elf_content = read_file(elf_filename)
>> +local elf_content = require('utils').tools.read_file(elf_filename)
> Nit: s/require('utils')/utils/
> Rationale: `utils` are already required above.
Fixed.
>
>> assert(#elf_content ~= 0, 'cannot read an object file')
>>
>> local strtab, symtab = read_elf(elf_content)
>> diff --git a/test/tarantool-tests/utils/tools.lua b/test/tarantool-tests/utils/tools.lua
>> index f35c6922..a2556e32 100644
>> --- a/test/tarantool-tests/utils/tools.lua
>> +++ b/test/tarantool-tests/utils/tools.lua
>> @@ -12,4 +12,12 @@ function M.profilename(name)
>> return (arg[0]:gsub('^(.+)/([^/]+)%.test%.lua$', replacepattern))
>> end
>>
>> +-- Reads a file located at a specified path and returns its content.
> Nit: Comment line width is more than 66 symbols.
Fixed.
>
>> +function M.read_file(path)
>> + local file = assert(io.open(path), 'cannot open an object file')
>> + local content =file:read('*a')
>> +file:close()
>> + return content
>> +end
>> +
>> return M
>> --
>> 2.34.1
>>
[-- Attachment #2: Type: text/html, Size: 3510 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Tarantool-patches] [PATCH luajit 3/4][v2] OSX/iOS/ARM64: Fix generation of Mach-O object files.
2024-04-11 13:22 [Tarantool-patches] [PATCH luajit 0/4][v2] Mach-O generation fixes Sergey Bronnikov via Tarantool-patches
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 1/4][v2] ci: add a workflow for testing with AVX512 Sergey Bronnikov via Tarantool-patches
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 2/4][v2] test: introduce a helper read_file Sergey Bronnikov via Tarantool-patches
@ 2024-04-11 13:22 ` Sergey Bronnikov via Tarantool-patches
2024-04-12 11:19 ` Sergey Kaplun via Tarantool-patches
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 4/4][v2] OSX/iOS/ARM64: Fix bytecode embedding in Mach-O object file Sergey Bronnikov via Tarantool-patches
3 siblings, 1 reply; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2024-04-11 13:22 UTC (permalink / raw)
To: tarantool-patches, Sergey Kaplun, Maxim Kokryashkin
From: Mike Pall <mike>
Thanks to Carlo Cabrera.
(cherry picked from commit 3065c910ad6027031aabe2dfd3c26a3d0f014b4f)
The Mach-O FAT object constructed by LuaJIT had an incorrect format.
The problem is reproduced when the target hardware platform has
AVX512F and LuaJIT is compiled with enabled AVX512F instructions.
The problem arises because LuaJIT FFI code for Mach-O file
generation in `bcsave.lua` relies on undefined behavior for
conversions to `uint32_t`. AVX512F has the `vcvttsd2usi`
instruction, which converts `double`/`float` to `uint32_t/uint64_t`.
Earlier architectures (SSE2, AVX2) are sorely lacking such
an instruction, as they only support signed conversions. Unsigned
conversions are done with a signed convert and range shifting -
the exact algorithm depends on the compiler. A side-effect of these
workarounds is that negative `double`/`float` often inadvertently
convert 'as expected', even though this is invoking undefined
behavior. Whereas `vcvttsd2usi` always returns 0x80000000 or
0x8000000000000000 for out-of-range inputs.
The patch fixes the problem, however, the real issue remains unfixed.
Sergey Bronnikov:
* added the description, the test for the problem and
flavor with AVX512 to `exotic-builds-testing` workflow
Part of tarantool/tarantool#9595
---
src/jit/bcsave.lua | 6 +-
test/LuaJIT-tests/CMakeLists.txt | 9 +
...-865-cross-generation-mach-o-file.test.lua | 300 ++++++++++++++++++
3 files changed, 312 insertions(+), 3 deletions(-)
create mode 100644 test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua
index a287d675..7aec1555 100644
--- a/src/jit/bcsave.lua
+++ b/src/jit/bcsave.lua
@@ -446,18 +446,18 @@ typedef struct {
uint32_t value;
} mach_nlist;
typedef struct {
- uint32_t strx;
+ int32_t strx;
uint8_t type, sect;
uint16_t desc;
uint64_t value;
} mach_nlist_64;
typedef struct
{
- uint32_t magic, nfat_arch;
+ int32_t magic, nfat_arch;
} mach_fat_header;
typedef struct
{
- uint32_t cputype, cpusubtype, offset, size, align;
+ int32_t cputype, cpusubtype, offset, size, align;
} mach_fat_arch;
typedef struct {
struct {
diff --git a/test/LuaJIT-tests/CMakeLists.txt b/test/LuaJIT-tests/CMakeLists.txt
index b8e4dfc4..6d073700 100644
--- a/test/LuaJIT-tests/CMakeLists.txt
+++ b/test/LuaJIT-tests/CMakeLists.txt
@@ -52,6 +52,15 @@ if(LUAJIT_NO_UNWIND)
set(LUAJIT_TEST_TAGS_EXTRA +internal_unwinder)
endif()
+if(CMAKE_C_FLAGS MATCHES "-march=skylake-avx512")
+ # FIXME: Test <bit64.lua> verifies bitwise operations on numbers.
+ # There is a known issue - bitop doesn't work in LuaJIT built
+ # with the enabled AVX512 instruction set, see
+ # https://github.com/tarantool/tarantool/issues/6787.
+ # Hence, skip this when "skylake-avx512" is passed.
+ set(LUAJIT_TEST_TAGS_EXTRA +avx512)
+endif()
+
set(TEST_SUITE_NAME "LuaJIT-tests")
# XXX: The call produces both test and target <LuaJIT-tests-deps>
diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
new file mode 100644
index 00000000..04fb5495
--- /dev/null
+++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
@@ -0,0 +1,300 @@
+local tap = require('tap')
+local test = tap.test('lj-865-cross-generation-mach-o-file')
+local utils = require('utils')
+
+test:plan(1)
+
+-- The test creates an object file in Mach-O format with LuaJIT
+-- bytecode and checks the validity of the object file fields.
+--
+-- The original problem is reproduced with LuaJIT that built with
+-- enabled AVX512F instructions. The support for AVX512F could be
+-- checked in `/proc/cpuinfo` on Linux and
+-- `sysctl hw.optional.avx512f` on Mac. AVX512F must be
+-- implicitly enabled in a C compiler by passing a CPU codename.
+-- Please take a look at the GCC Online Documentation [1] for
+-- available CPU codenames. Also, see the Wikipedia for CPUs with
+-- AVX-512 support [2].
+-- To detect the CPU codename execute:
+-- `gcc -march=native -Q --help=target | grep march`.
+--
+-- 1. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html
+-- 2. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512
+--
+-- Manual steps for reproducing are the following:
+--
+-- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build
+-- $ cmake --build build --parallel
+-- $ echo > test.lua
+-- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o
+-- $ file test.o
+-- empty.o: DOS executable (block device driver)
+
+local ffi = require('ffi')
+
+-- LuaJIT can generate so called Universal Binary with Lua
+-- bytecode. The Universal Binary format is a format for
+-- executable files that run natively on hardware platforms with
+-- different hardware architectures. This concept is more
+-- generally known as a fat binary.
+--
+-- The format of the Mach-O is described in the document
+-- "OS X ABI Mach-O File Format Reference", published by Apple
+-- company. The copy of the (now removed) official documentation
+-- can be found here [1]. Yet another source of truth is
+-- XNU headers, see the definition of C-structures in:
+-- [2] (`nlist_64`), [3] (`fat_arch` and `fat_header`).
+--
+-- There are a good visual representation of Universal Binary
+-- in "Mac OS X Internals" book (pages 67-68) [5] and in the [6].
+-- Below is the schematic structure of Universal Binary, which
+-- includes two executables for PowerPC and Intel i386 (omitted):
+--
+-- 0x0000000 ---------------------------------------
+-- |
+-- struct | 0xcafebabe FAT_MAGIC magic
+-- fat_header | -------------------------------------
+-- | 0x00000003 nfat_arch
+-- ---------------------------------------
+-- | 0x00000012 CPU_TYPE_POWERPC cputype
+-- | -------------------------------------
+-- | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype
+-- struct | -------------------------------------
+-- fat_arch | 0x00001000 4096 bytes offset
+-- | -------------------------------------
+-- | 0x00004224 16932 bytes size
+-- | -------------------------------------
+-- | 0x0000000c 2^12 = 4096 bytes align
+-- ---------------------------------------
+-- ---------------------------------------
+-- | 0x00000007 CPU_TYPE_I386 cputype
+-- | -------------------------------------
+-- | 0x00000003 CPU_SUBTYPE_I386_ALL cpusubtype
+-- struct | -------------------------------------
+-- fat_arch | 0x00006000 24576 bytes offset
+-- | -------------------------------------
+-- | 0x0000292c 10540 bytes size
+-- | -------------------------------------
+-- | 0x0000000c 2^12 = 4096 bytes align
+-- ---------------------------------------
+-- Unused
+-- 0x00001000 ---------------------------------------
+-- | 0xfeedface MH_MAGIC magic
+-- | ------------------------------------
+-- | 0x00000012 CPU_TYPE_POWERPC cputype
+-- | ------------------------------------
+-- struct | 0x00000000 CPU_SUBTYPE_POWERPC_ALL cpusubtype
+-- mach_header | ------------------------------------
+-- | 0x00000002 MH_EXECUTE filetype
+-- | ------------------------------------
+-- | 0x0000000b 10 load commands ncmds
+-- | ------------------------------------
+-- | 0x00000574 1396 bytes sizeofcmds
+-- | ------------------------------------
+-- | 0x00000085 DYLDLINK TWOLEVEL flags
+-- --------------------------------------
+-- Load commands
+-- ---------------------------------------
+-- Data
+-- ---------------------------------------
+--
+-- < x86 executable >
+--
+-- 1. https://github.com/aidansteele/osx-abi-macho-file-format-reference
+-- 2. https://github.com/apple-oss-distributions/xnu/blob/xnu-10002.1.13/EXTERNAL_HEADERS/mach-o/nlist.h
+-- 3. https://github.com/apple-oss-distributions/xnu/blob/xnu-10002.1.13/EXTERNAL_HEADERS/mach-o/fat.h
+-- 4. https://developer.apple.com/documentation/apple-silicon/addressing-architectural-differences-in-your-macos-code
+-- 5. https://reverseengineering.stackexchange.com/a/6357/46029
+-- 6. http://formats.kaitai.io/mach_o/index.html
+--
+-- Using the same declarations as defined in <src/jit/bcsave.lua>.
+ffi.cdef[[
+typedef struct
+{
+ uint32_t magic, cputype, cpusubtype, filetype, ncmds, sizeofcmds, flags;
+} mach_header;
+
+typedef struct {
+ uint32_t cmd, cmdsize;
+ char segname[16];
+ uint32_t vmaddr, vmsize, fileoff, filesize;
+ uint32_t maxprot, initprot, nsects, flags;
+} mach_segment_command;
+
+typedef struct {
+ char sectname[16], segname[16];
+ uint32_t addr, size;
+ uint32_t offset, align, reloff, nreloc, flags;
+ uint32_t reserved1, reserved2;
+} mach_section;
+
+typedef struct {
+ uint32_t cmd, cmdsize, symoff, nsyms, stroff, strsize;
+} mach_symtab_command;
+
+typedef struct {
+ int32_t strx;
+ uint8_t type, sect;
+ int16_t desc;
+ uint32_t value;
+} mach_nlist;
+
+typedef struct
+{
+ int32_t magic, nfat_arch;
+} mach_fat_header;
+
+typedef struct
+{
+ int32_t cputype, cpusubtype, offset, size, align;
+} mach_fat_arch;
+
+typedef struct {
+ mach_fat_header fat;
+ mach_fat_arch fat_arch[2];
+ struct {
+ mach_header hdr;
+ mach_segment_command seg;
+ mach_section sec;
+ mach_symtab_command sym;
+ } arch[2];
+ mach_nlist sym_entry;
+ uint8_t space[4096];
+} mach_fat_obj;
+]]
+
+local function create_obj_file(name, arch)
+ local mach_o_path = os.tmpname() .. '.o'
+ local lua_path = os.getenv('LUA_PATH')
+ local lua_bin = utils.exec.luacmd(arg):match('%S+')
+ local cmd_fmt = 'LUA_PATH="%s" %s -b -n "%s" -o osx -a %s -e "print()" %s'
+ local cmd = (cmd_fmt):format(lua_path, lua_bin, name, arch, mach_o_path)
+ local ret = os.execute(cmd)
+ assert(ret == 0, 'cannot create an object file')
+ return mach_o_path
+end
+
+-- Parses a buffer in the Mach-O format and returns
+-- the FAT magic number and `nfat_arch`.
+local function read_mach_o(buf)
+ local res = {
+ header = {
+ magic = 0,
+ nfat_arch = 0,
+ },
+ fat_arch = {},
+ }
+
+ -- Mach-O FAT object.
+ local mach_fat_obj_type = ffi.typeof('mach_fat_obj *')
+ local obj = ffi.cast(mach_fat_obj_type, buf)
+
+ -- Mach-O FAT object header.
+ local mach_fat_header = obj.fat
+ -- Mach-O FAT is BE, target arch is LE.
+ local be32 = bit.bswap
+ res.header.magic = be32(mach_fat_header.magic)
+ res.header.nfat_arch = be32(mach_fat_header.nfat_arch)
+
+ -- Mach-O FAT object arches.
+ for i = 0, res.header.nfat_arch - 1 do
+ local fat_arch = obj.fat_arch[i]
+ local arch = {
+ cputype = be32(fat_arch.cputype),
+ cpusubtype = be32(fat_arch.cpusubtype),
+ }
+ table.insert(res.fat_arch, arch)
+ end
+
+ return res
+end
+
+-- Universal Binary can contain executables for more than one
+-- CPU architecture. For simplicity, the test compares the sum of
+-- CPU types and CPU subtypes.
+--
+-- <src/jit/bcsave.lua:bcsave_machobj> has the definitions of the
+-- numbers below. The original XNU source code may be found in
+-- <osfmk/mach/machine.h> [1].
+--
+-- 1. https://opensource.apple.com/source/xnu/xnu-4570.41.2/osfmk/mach/machine.h.auto.html
+--
+local SUM_CPUTYPE = {
+ arm = 7 + 12,
+}
+local SUM_CPUSUBTYPE = {
+ arm = 3 + 9,
+}
+
+-- The function builds Mach-O FAT object file and retrieves
+-- its header fields (magic and nfat_arch) and fields of each arch
+-- (cputype, cpusubtype).
+--
+-- The Mach-O FAT object header can be retrieved with `otool` on
+-- macOS:
+--
+-- $ otool -f empty.o
+-- Fat headers
+-- fat_magic 0xcafebabe
+-- nfat_arch 2
+-- <snipped>
+--
+-- CPU type and subtype can be retrieved with `lipo` on macOS:
+--
+-- $ luajit -b -o osx -a arm empty.lua empty.o
+-- $ lipo -archs empty.o
+-- i386 armv7
+-- $ luajit -b -o osx -a arm64 empty.lua empty.o
+-- $ lipo -archs empty.o
+-- x86_64 arm64
+local function build_and_check_mach_o(subtest, hw_arch)
+ assert(hw_arch == 'arm')
+
+ subtest:plan(4)
+ -- FAT_MAGIC is an integer containing the value 0xCAFEBABE in
+ -- big-endian byte order format. On a big-endian host CPU,
+ -- this can be validated using the constant FAT_MAGIC;
+ -- on a little-endian host CPU, it can be validated using
+ -- the constant FAT_CIGAM.
+ --
+ -- FAT_NARCH is an integer specifying the number of fat_arch
+ -- data structures that follow. This is the number of
+ -- architectures contained in this binary.
+ --
+ -- See the aforementioned "OS X ABI Mach-O File Format
+ -- Reference".
+ local FAT_MAGIC = '0xffffffffcafebabe'
+ local FAT_NARCH = 2
+
+ local MODULE_NAME = 'lango_team'
+
+ local mach_o_obj_path = create_obj_file(MODULE_NAME, hw_arch)
+ local mach_o_buf = utils.tools.read_file(mach_o_obj_path)
+ assert(mach_o_buf ~= nil and #mach_o_buf ~= 0, 'cannot read an object file')
+
+ local mach_o = read_mach_o(mach_o_buf)
+
+ -- Teardown.
+ assert(os.remove(mach_o_obj_path), 'remove an object file')
+
+ local magic_str = string.format('%#x', mach_o.header.magic)
+ subtest:is(magic_str, FAT_MAGIC,
+ 'fat_magic is correct in Mach-O')
+ subtest:is(mach_o.header.nfat_arch, FAT_NARCH,
+ 'nfat_arch is correct in Mach-O')
+
+ local total_cputype = 0
+ local total_cpusubtype = 0
+ for i = 1, FAT_NARCH do
+ total_cputype = total_cputype + mach_o.fat_arch[i].cputype
+ total_cpusubtype = total_cpusubtype + mach_o.fat_arch[i].cpusubtype
+ end
+ subtest:is(total_cputype, SUM_CPUTYPE[hw_arch],
+ 'cputype is correct in Mach-O')
+ subtest:is(total_cpusubtype, SUM_CPUSUBTYPE[hw_arch],
+ 'cpusubtype is correct in Mach-O')
+end
+
+test:test('arm', build_and_check_mach_o, 'arm')
+
+test:done(true)
--
2.34.1
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Tarantool-patches] [PATCH luajit 3/4][v2] OSX/iOS/ARM64: Fix generation of Mach-O object files.
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 3/4][v2] OSX/iOS/ARM64: Fix generation of Mach-O object files Sergey Bronnikov via Tarantool-patches
@ 2024-04-12 11:19 ` Sergey Kaplun via Tarantool-patches
2024-04-16 15:29 ` Sergey Bronnikov via Tarantool-patches
0 siblings, 1 reply; 14+ messages in thread
From: Sergey Kaplun via Tarantool-patches @ 2024-04-12 11:19 UTC (permalink / raw)
To: Sergey Bronnikov; +Cc: tarantool-patches
Hi, Sergey!
Thanks for the fixes!
LGTM after fixing a few minor nits below.
On 11.04.24, Sergey Bronnikov wrote:
> From: Mike Pall <mike>
>
<snipped>
> ---
> src/jit/bcsave.lua | 6 +-
> test/LuaJIT-tests/CMakeLists.txt | 9 +
> ...-865-cross-generation-mach-o-file.test.lua | 300 ++++++++++++++++++
> 3 files changed, 312 insertions(+), 3 deletions(-)
> create mode 100644 test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
>
> diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua
> index a287d675..7aec1555 100644
> --- a/src/jit/bcsave.lua
> +++ b/src/jit/bcsave.lua
<snipped>
> diff --git a/test/LuaJIT-tests/CMakeLists.txt b/test/LuaJIT-tests/CMakeLists.txt
> index b8e4dfc4..6d073700 100644
> --- a/test/LuaJIT-tests/CMakeLists.txt
> +++ b/test/LuaJIT-tests/CMakeLists.txt
> @@ -52,6 +52,15 @@ if(LUAJIT_NO_UNWIND)
> set(LUAJIT_TEST_TAGS_EXTRA +internal_unwinder)
> endif()
>
> +if(CMAKE_C_FLAGS MATCHES "-march=skylake-avx512")
> + # FIXME: Test <bit64.lua> verifies bitwise operations on numbers.
Nit: comment line width is more than 66 symbols.
> + # There is a known issue - bitop doesn't work in LuaJIT built
> + # with the enabled AVX512 instruction set, see
> + # https://github.com/tarantool/tarantool/issues/6787.
> + # Hence, skip this when "skylake-avx512" is passed.
> + set(LUAJIT_TEST_TAGS_EXTRA +avx512)
> +endif()
> +
Should this be a part of the first commit?
> set(TEST_SUITE_NAME "LuaJIT-tests")
>
> # XXX: The call produces both test and target <LuaJIT-tests-deps>
> diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
> new file mode 100644
> index 00000000..04fb5495
> --- /dev/null
> +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
> @@ -0,0 +1,300 @@
> +local tap = require('tap')
> +local test = tap.test('lj-865-cross-generation-mach-o-file')
> +local utils = require('utils')
> +
> +test:plan(1)
> +
> +-- The test creates an object file in Mach-O format with LuaJIT
> +-- bytecode and checks the validity of the object file fields.
> +--
> +-- The original problem is reproduced with LuaJIT that built with
Typo: s/LuaJIT that built/LuaJIT, which is built/
> +-- enabled AVX512F instructions. The support for AVX512F could be
> +-- checked in `/proc/cpuinfo` on Linux and
> +-- `sysctl hw.optional.avx512f` on Mac. AVX512F must be
> +-- implicitly enabled in a C compiler by passing a CPU codename.
> +-- Please take a look at the GCC Online Documentation [1] for
> +-- available CPU codenames. Also, see the Wikipedia for CPUs with
> +-- AVX-512 support [2].
> +-- To detect the CPU codename execute:
Typo: s/codename/codename,/
> +-- `gcc -march=native -Q --help=target | grep march`.
> +--
> +-- 1. https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html
> +-- 2. https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512
> +--
> +-- Manual steps for reproducing are the following:
> +--
> +-- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build
> +-- $ cmake --build build --parallel
> +-- $ echo > test.lua
> +-- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o
> +-- $ file test.o
> +-- empty.o: DOS executable (block device driver)
> +
> +local ffi = require('ffi')
Nit: Why do we require the ffi here alongside from others requires?
> +
> +-- LuaJIT can generate so called Universal Binary with Lua
<snipped>
> +--
> +-- There are a good visual representation of Universal Binary
Typo: s/are/is/
> +-- in "Mac OS X Internals" book (pages 67-68) [5] and in the [6].
> +-- Below is the schematic structure of Universal Binary, which
> +-- includes two executables for PowerPC and Intel i386 (omitted):
> +--
> +-- 0x0000000 ---------------------------------------
<snipped>
> +local function create_obj_file(name, arch)
> + local mach_o_path = os.tmpname() .. '.o'
> + local lua_path = os.getenv('LUA_PATH')
> + local lua_bin = utils.exec.luacmd(arg):match('%S+')
> + local cmd_fmt = 'LUA_PATH="%s" %s -b -n "%s" -o osx -a %s -e "print()" %s'
> + local cmd = (cmd_fmt):format(lua_path, lua_bin, name, arch, mach_o_path)
Nit: Typo: s/(cmd_fmt)/cmd_fmt/
> + local ret = os.execute(cmd)
> + assert(ret == 0, 'cannot create an object file')
> + return mach_o_path
> +end
> +
> +-- Parses a buffer in the Mach-O format and returns
Nit: The comment line looks underfilled.
> +-- the FAT magic number and `nfat_arch`.
> +local function read_mach_o(buf)
<snipped>
> +local SUM_CPUTYPE = {
Minor: It will be nice to add the comment:
| -- x86 + arm.
> + arm = 7 + 12,
> +}
> +local SUM_CPUSUBTYPE = {
Minor: It will be nice to add the comment:
| -- x86 + arm.
> + arm = 3 + 9,
> +}
> +
<snipped>
> +local function build_and_check_mach_o(subtest, hw_arch)
> + assert(hw_arch == 'arm')
> +
> + subtest:plan(4)
<snipped>
> +test:test('arm', build_and_check_mach_o, 'arm')
Minor: we can use `subtest.name` as the definition of the `hw_arch` in the
`build_and_check_mach_o()`, so it helps to avoid duplication of arch
usage.
Matter of taste.
Feel free to ignore.
> +
> +test:done(true)
> --
> 2.34.1
>
--
Best regards,
Sergey Kaplun
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Tarantool-patches] [PATCH luajit 3/4][v2] OSX/iOS/ARM64: Fix generation of Mach-O object files.
2024-04-12 11:19 ` Sergey Kaplun via Tarantool-patches
@ 2024-04-16 15:29 ` Sergey Bronnikov via Tarantool-patches
0 siblings, 0 replies; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2024-04-16 15:29 UTC (permalink / raw)
To: Sergey Kaplun, Sergey Bronnikov; +Cc: tarantool-patches
[-- Attachment #1: Type: text/plain, Size: 5571 bytes --]
Hi, Sergey
On 12.04.2024 14:19, Sergey Kaplun via Tarantool-patches wrote:
> Hi, Sergey!
> Thanks for the fixes!
> LGTM after fixing a few minor nits below.
>
> On 11.04.24, Sergey Bronnikov wrote:
>> From: Mike Pall <mike>
>>
> <snipped>
>
>> ---
>> src/jit/bcsave.lua | 6 +-
>> test/LuaJIT-tests/CMakeLists.txt | 9 +
>> ...-865-cross-generation-mach-o-file.test.lua | 300 ++++++++++++++++++
>> 3 files changed, 312 insertions(+), 3 deletions(-)
>> create mode 100644 test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
>>
>> diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua
>> index a287d675..7aec1555 100644
>> --- a/src/jit/bcsave.lua
>> +++ b/src/jit/bcsave.lua
> <snipped>
>
>> diff --git a/test/LuaJIT-tests/CMakeLists.txt b/test/LuaJIT-tests/CMakeLists.txt
>> index b8e4dfc4..6d073700 100644
>> --- a/test/LuaJIT-tests/CMakeLists.txt
>> +++ b/test/LuaJIT-tests/CMakeLists.txt
>> @@ -52,6 +52,15 @@ if(LUAJIT_NO_UNWIND)
>> set(LUAJIT_TEST_TAGS_EXTRA +internal_unwinder)
>> endif()
>>
>> +if(CMAKE_C_FLAGS MATCHES "-march=skylake-avx512")
>> + # FIXME: Test <bit64.lua> verifies bitwise operations on numbers.
> Nit: comment line width is more than 66 symbols.
Fixed.
>> + # There is a known issue - bitop doesn't work in LuaJIT built
>> + # with the enabled AVX512 instruction set, see
>> + #https://github.com/tarantool/tarantool/issues/6787.
>> + # Hence, skip this when "skylake-avx512" is passed.
>> + set(LUAJIT_TEST_TAGS_EXTRA +avx512)
>> +endif()
>> +
> Should this be a part of the first commit?
Moved to the first commit.
>
>> set(TEST_SUITE_NAME "LuaJIT-tests")
>>
>> # XXX: The call produces both test and target <LuaJIT-tests-deps>
>> diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
>> new file mode 100644
>> index 00000000..04fb5495
>> --- /dev/null
>> +++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
>> @@ -0,0 +1,300 @@
>> +local tap = require('tap')
>> +local test = tap.test('lj-865-cross-generation-mach-o-file')
>> +local utils = require('utils')
>> +
>> +test:plan(1)
>> +
>> +-- The test creates an object file in Mach-O format with LuaJIT
>> +-- bytecode and checks the validity of the object file fields.
>> +--
>> +-- The original problem is reproduced with LuaJIT that built with
> Typo: s/LuaJIT that built/LuaJIT, which is built/
Fixed.
>
>> +-- enabled AVX512F instructions. The support for AVX512F could be
>> +-- checked in `/proc/cpuinfo` on Linux and
>> +-- `sysctl hw.optional.avx512f` on Mac. AVX512F must be
>> +-- implicitly enabled in a C compiler by passing a CPU codename.
>> +-- Please take a look at the GCC Online Documentation [1] for
>> +-- available CPU codenames. Also, see the Wikipedia for CPUs with
>> +-- AVX-512 support [2].
>> +-- To detect the CPU codename execute:
> Typo: s/codename/codename,/
Fixed.
>
>> +-- `gcc -march=native -Q --help=target | grep march`.
>> +--
>> +-- 1.https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html
>> +-- 2.https://en.wikipedia.org/wiki/AVX-512#CPUs_with_AVX-512
>> +--
>> +-- Manual steps for reproducing are the following:
>> +--
>> +-- $ CC=gcc TARGET_CFLAGS='skylake-avx512' cmake -S . -B build
>> +-- $ cmake --build build --parallel
>> +-- $ echo > test.lua
>> +-- $ LUA_PATH="src/?.lua;;" luajit -b -o osx -a arm test.lua test.o
>> +-- $ file test.o
>> +-- empty.o: DOS executable (block device driver)
>> +
>> +local ffi = require('ffi')
> Nit: Why do we require the ffi here alongside from others requires?
Moved to another place.
>> +
>> +-- LuaJIT can generate so called Universal Binary with Lua
> <snipped>
>
>> +--
>> +-- There are a good visual representation of Universal Binary
> Typo: s/are/is/
Fixed.
>
>> +-- in "Mac OS X Internals" book (pages 67-68) [5] and in the [6].
>> +-- Below is the schematic structure of Universal Binary, which
>> +-- includes two executables for PowerPC and Intel i386 (omitted):
>> +--
>> +-- 0x0000000 ---------------------------------------
> <snipped>
>
>> +local function create_obj_file(name, arch)
>> + local mach_o_path = os.tmpname() .. '.o'
>> + local lua_path = os.getenv('LUA_PATH')
>> + local lua_bin = utils.exec.luacmd(arg):match('%S+')
>> + local cmd_fmt = 'LUA_PATH="%s" %s -b -n "%s" -o osx -a %s -e "print()" %s'
>> + local cmd = (cmd_fmt):format(lua_path, lua_bin, name, arch, mach_o_path)
> Nit: Typo: s/(cmd_fmt)/cmd_fmt/
Fixed.
>
>> + local ret = os.execute(cmd)
>> + assert(ret == 0, 'cannot create an object file')
>> + return mach_o_path
>> +end
>> +
>> +-- Parses a buffer in the Mach-O format and returns
> Nit: The comment line looks underfilled.
Refilled it.
>
>> +-- the FAT magic number and `nfat_arch`.
>> +local function read_mach_o(buf)
> <snipped>
>
>> +local SUM_CPUTYPE = {
> Minor: It will be nice to add the comment:
> | -- x86 + arm.
Added a comment
>> + arm = 7 + 12,
>> +}
>> +local SUM_CPUSUBTYPE = {
> Minor: It will be nice to add the comment:
> | -- x86 + arm.
>
>> + arm = 3 + 9,
>> +}
>> +
> <snipped>
>
>> +local function build_and_check_mach_o(subtest, hw_arch)
>> + assert(hw_arch == 'arm')
>> +
>> + subtest:plan(4)
> <snipped>
>
>> +test:test('arm', build_and_check_mach_o, 'arm')
> Minor: we can use `subtest.name` as the definition of the `hw_arch` in the
> `build_and_check_mach_o()`, so it helps to avoid duplication of arch
> usage.
>
> Matter of taste.
> Feel free to ignore.
ignored
>> +
>> +test:done(true)
>> --
>> 2.34.1
>>
[-- Attachment #2: Type: text/html, Size: 10315 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* [Tarantool-patches] [PATCH luajit 4/4][v2] OSX/iOS/ARM64: Fix bytecode embedding in Mach-O object file.
2024-04-11 13:22 [Tarantool-patches] [PATCH luajit 0/4][v2] Mach-O generation fixes Sergey Bronnikov via Tarantool-patches
` (2 preceding siblings ...)
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 3/4][v2] OSX/iOS/ARM64: Fix generation of Mach-O object files Sergey Bronnikov via Tarantool-patches
@ 2024-04-11 13:22 ` Sergey Bronnikov via Tarantool-patches
2024-04-12 11:27 ` Sergey Kaplun via Tarantool-patches
3 siblings, 1 reply; 14+ messages in thread
From: Sergey Bronnikov via Tarantool-patches @ 2024-04-11 13:22 UTC (permalink / raw)
To: tarantool-patches, Sergey Kaplun, Maxim Kokryashkin
From: Mike Pall <mike>
Thanks to Carlo Cabrera.
(cherry picked from commit b98b37231bd2dcb79e10b0f974cefd91eb0d7b3a)
Mach-O FAT object files generated by LuaJIT for ARM64 Had
an incorrect format due to the usage of the 32-bit version of
FFI structure. This patch adds the 64-bit structure definition
and uses it for ARM64.
Sergey Bronnikov:
* added the description and the test for the problem
Part of tarantool/tarantool#9595
---
src/jit/bcsave.lua | 14 ++++-
...-865-cross-generation-mach-o-file.test.lua | 55 +++++++++++++++++--
2 files changed, 63 insertions(+), 6 deletions(-)
diff --git a/src/jit/bcsave.lua b/src/jit/bcsave.lua
index 7aec1555..069cf1a3 100644
--- a/src/jit/bcsave.lua
+++ b/src/jit/bcsave.lua
@@ -491,6 +491,18 @@ typedef struct {
mach_nlist sym_entry;
uint8_t space[4096];
} mach_fat_obj;
+typedef struct {
+ mach_fat_header fat;
+ mach_fat_arch fat_arch[2];
+ struct {
+ mach_header_64 hdr;
+ mach_segment_command_64 seg;
+ mach_section_64 sec;
+ mach_symtab_command sym;
+ } arch[2];
+ mach_nlist_64 sym_entry;
+ uint8_t space[4096];
+} mach_fat_obj_64;
]]
local symname = '_'..LJBC_PREFIX..ctx.modname
local isfat, is64, align, mobj = false, false, 4, "mach_obj"
@@ -499,7 +511,7 @@ typedef struct {
elseif ctx.arch == "arm" then
isfat, mobj = true, "mach_fat_obj"
elseif ctx.arch == "arm64" then
- is64, align, isfat, mobj = true, 8, true, "mach_fat_obj"
+ is64, align, isfat, mobj = true, 8, true, "mach_fat_obj_64"
else
check(ctx.arch == "x86", "unsupported architecture for OSX")
end
diff --git a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
index 04fb5495..07988948 100644
--- a/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
+++ b/test/tarantool-tests/lj-865-cross-generation-mach-o-file.test.lua
@@ -2,7 +2,7 @@ local tap = require('tap')
local test = tap.test('lj-865-cross-generation-mach-o-file')
local utils = require('utils')
-test:plan(1)
+test:plan(2)
-- The test creates an object file in Mach-O format with LuaJIT
-- bytecode and checks the validity of the object file fields.
@@ -114,6 +114,11 @@ typedef struct
uint32_t magic, cputype, cpusubtype, filetype, ncmds, sizeofcmds, flags;
} mach_header;
+typedef struct
+{
+ mach_header; uint32_t reserved;
+} mach_header_64;
+
typedef struct {
uint32_t cmd, cmdsize;
char segname[16];
@@ -121,6 +126,13 @@ typedef struct {
uint32_t maxprot, initprot, nsects, flags;
} mach_segment_command;
+typedef struct {
+ uint32_t cmd, cmdsize;
+ char segname[16];
+ uint64_t vmaddr, vmsize, fileoff, filesize;
+ uint32_t maxprot, initprot, nsects, flags;
+} mach_segment_command_64;
+
typedef struct {
char sectname[16], segname[16];
uint32_t addr, size;
@@ -128,6 +140,13 @@ typedef struct {
uint32_t reserved1, reserved2;
} mach_section;
+typedef struct {
+ char sectname[16], segname[16];
+ uint64_t addr, size;
+ uint32_t offset, align, reloff, nreloc, flags;
+ uint32_t reserved1, reserved2, reserved3;
+} mach_section_64;
+
typedef struct {
uint32_t cmd, cmdsize, symoff, nsyms, stroff, strsize;
} mach_symtab_command;
@@ -139,6 +158,13 @@ typedef struct {
uint32_t value;
} mach_nlist;
+typedef struct {
+ int32_t strx;
+ uint8_t type, sect;
+ uint16_t desc;
+ uint64_t value;
+} mach_nlist_64;
+
typedef struct
{
int32_t magic, nfat_arch;
@@ -161,6 +187,19 @@ typedef struct {
mach_nlist sym_entry;
uint8_t space[4096];
} mach_fat_obj;
+
+typedef struct {
+ mach_fat_header fat;
+ mach_fat_arch fat_arch[2];
+ struct {
+ mach_header_64 hdr;
+ mach_segment_command_64 seg;
+ mach_section_64 sec;
+ mach_symtab_command sym;
+ } arch[2];
+ mach_nlist_64 sym_entry;
+ uint8_t space[4096];
+} mach_fat_obj_64;
]]
local function create_obj_file(name, arch)
@@ -176,7 +215,7 @@ end
-- Parses a buffer in the Mach-O format and returns
-- the FAT magic number and `nfat_arch`.
-local function read_mach_o(buf)
+local function read_mach_o(buf, hw_arch)
local res = {
header = {
magic = 0,
@@ -185,8 +224,11 @@ local function read_mach_o(buf)
fat_arch = {},
}
+ local is64 = hw_arch == 'arm64'
+
-- Mach-O FAT object.
- local mach_fat_obj_type = ffi.typeof('mach_fat_obj *')
+ local mach_fat_obj_type = ffi.typeof(is64 and
+ 'mach_fat_obj_64 *' or 'mach_fat_obj *')
local obj = ffi.cast(mach_fat_obj_type, buf)
-- Mach-O FAT object header.
@@ -221,9 +263,11 @@ end
--
local SUM_CPUTYPE = {
arm = 7 + 12,
+ arm64 = 0x01000007 + 0x0100000c,
}
local SUM_CPUSUBTYPE = {
arm = 3 + 9,
+ arm64 = 3 + 0,
}
-- The function builds Mach-O FAT object file and retrieves
@@ -248,7 +292,7 @@ local SUM_CPUSUBTYPE = {
-- $ lipo -archs empty.o
-- x86_64 arm64
local function build_and_check_mach_o(subtest, hw_arch)
- assert(hw_arch == 'arm')
+ assert(hw_arch == 'arm' or hw_arch == 'arm64')
subtest:plan(4)
-- FAT_MAGIC is an integer containing the value 0xCAFEBABE in
@@ -272,7 +316,7 @@ local function build_and_check_mach_o(subtest, hw_arch)
local mach_o_buf = utils.tools.read_file(mach_o_obj_path)
assert(mach_o_buf ~= nil and #mach_o_buf ~= 0, 'cannot read an object file')
- local mach_o = read_mach_o(mach_o_buf)
+ local mach_o = read_mach_o(mach_o_buf, hw_arch)
-- Teardown.
assert(os.remove(mach_o_obj_path), 'remove an object file')
@@ -296,5 +340,6 @@ local function build_and_check_mach_o(subtest, hw_arch)
end
test:test('arm', build_and_check_mach_o, 'arm')
+test:test('arm64', build_and_check_mach_o, 'arm64')
test:done(true)
--
2.34.1
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Tarantool-patches] [PATCH luajit 4/4][v2] OSX/iOS/ARM64: Fix bytecode embedding in Mach-O object file.
2024-04-11 13:22 ` [Tarantool-patches] [PATCH luajit 4/4][v2] OSX/iOS/ARM64: Fix bytecode embedding in Mach-O object file Sergey Bronnikov via Tarantool-patches
@ 2024-04-12 11:27 ` Sergey Kaplun via Tarantool-patches
0 siblings, 0 replies; 14+ messages in thread
From: Sergey Kaplun via Tarantool-patches @ 2024-04-12 11:27 UTC (permalink / raw)
To: Sergey Bronnikov; +Cc: tarantool-patches
Hi, Sergey!
Thanks for the fixes!
LGTM, with a several minor nits below.
On 11.04.24, Sergey Bronnikov wrote:
> From: Mike Pall <mike>
>
> Thanks to Carlo Cabrera.
>
> (cherry picked from commit b98b37231bd2dcb79e10b0f974cefd91eb0d7b3a)
>
> Mach-O FAT object files generated by LuaJIT for ARM64 Had
Typo: s/Had/had/
> an incorrect format due to the usage of the 32-bit version of
> FFI structure. This patch adds the 64-bit structure definition
Typo: s/FFI structure/the FFI structure/
> and uses it for ARM64.
>
> Sergey Bronnikov:
> * added the description and the test for the problem
>
> Part of tarantool/tarantool#9595
> ---
> src/jit/bcsave.lua | 14 ++++-
> ...-865-cross-generation-mach-o-file.test.lua | 55 +++++++++++++++++--
> 2 files changed, 63 insertions(+), 6 deletions(-)
>
<snipped>
>
> -- Mach-O FAT object header.
> @@ -221,9 +263,11 @@ end
> --
> local SUM_CPUTYPE = {
> arm = 7 + 12,
> + arm64 = 0x01000007 + 0x0100000c,
Minor: The comment will be appreciated:
| x64 + arm64.
> }
> local SUM_CPUSUBTYPE = {
> arm = 3 + 9,
> + arm64 = 3 + 0,
Minor: The comment will be appreciated:
| x64 + arm64.
> }
>
> -- The function builds Mach-O FAT object file and retrieves
<snipped>
> --
> 2.34.1
>
--
Best regards,
Sergey Kaplun
^ permalink raw reply [flat|nested] 14+ messages in thread