On Thursday, 4 April 2024 at 13:15:50 UTC, Hipreme wrote:
> On Thursday, 4 April 2024 at 12:27:00 UTC, Paolo Invernizzi wrote:
>> On Thursday, 4 April 2024 at 11:10:04 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>>
>>> On 05/04/2024 12:07 AM, Paolo Invernizzi wrote:
>>>> [...]
>>>
>>> No I meant shared libraries.
>>>
>>> Specifically the distribution of the source files that act as the interface to it.
>>>
>>> That way you have the .sar file and the .dll and that's everything you need to use it.
>>
>> Aren't 'di' sources the target solution for library API? What's the problem in distributing a zip or tar?
>>
>> In C++ you usually have a specific "include" directory with all you need, what's the burden in doing a zip with the shared library binary and the 'di' directories?
>>
>> /P
>
> The .di feature is a big flop of D. They are nearly useless, their generation is dumb since no code analysis is done for removing useless imports. It should analyze the code and: check if the type import is used or not. Also, public imports are always kept.
> There's no reason nowadays to use auto DI generation. One must either completely ignore this feature or just create their own DI files.

Let's put aside the implementation of the automatic way to generate '.di', and rephrase:
Are .di intended to be the correct way to expose _public_ API of an opaque library binary?

What's the problem SAR targets to solve that a package manager can't solve? Why D needs it?

On 05/04/2024 3:11 AM, Paolo Invernizzi wrote:
> Let's put aside the implementation of the automatic way to generate '.di', and rephrase: Are .di intended to be the correct way to expose /public/ API of an opaque library binary?

I intend to see this happen yes.

If we don't do this, language features like inference simply make it impossible to link against code that has had it.

It will also mean shared libraries can hide their internal details, which is something I care about greatly.

On Thursday, 4 April 2024 at 14:11:35 UTC, Paolo Invernizzi wrote:
> On Thursday, 4 April 2024 at 13:15:50 UTC, Hipreme wrote:
>> On Thursday, 4 April 2024 at 12:27:00 UTC, Paolo Invernizzi wrote:
>>> On Thursday, 4 April 2024 at 11:10:04 UTC, Richard (Rikki) Andrew Cattermole wrote:
>>>>
>>>> On 05/04/2024 12:07 AM, Paolo Invernizzi wrote:
>>>>> [...]
>>>>
>>>> No I meant shared libraries.
>>>>
>>>> Specifically the distribution of the source files that act as the interface to it.
>>>>
>>>> That way you have the .sar file and the .dll and that's everything you need to use it.
>>>
>>> Aren't 'di' sources the target solution for library API? What's the problem in distributing a zip or tar?
>>>
>>> In C++ you usually have a specific "include" directory with all you need, what's the burden in doing a zip with the shared library binary and the 'di' directories?
>>>
>>> /P
>>
>> The .di feature is a big flop of D. They are nearly useless, their generation is dumb since no code analysis is done for removing useless imports. It should analyze the code and: check if the type import is used or not. Also, public imports are always kept.
>> There's no reason nowadays to use auto DI generation. One must either completely ignore this feature or just create their own DI files.
>
> Let's put aside the implementation of the automatic way to generate '.di', and rephrase:
> Are .di intended to be the correct way to expose _public_ API of an opaque library binary?
>
> What's the problem SAR targets to solve that a package manager can't solve? Why D needs it?


On the case of it being opaque, I could not care less. Yes, .di files are the correct way to expose public API. When they are used correctly, you could reduce by a lot the compilation time required by your files.

SAR solves exactly 0 things. It wasn't even put to test and Dennis even tested in the thread discussion SAR vs dmd -i. DMD -i was faster. Walter said that a cold run could be way faster though since the files won't need to be mem mapped.

On Thursday, 4 April 2024 at 15:36:31 UTC, Hipreme wrote:
> SAR solves exactly 0 things. It wasn't even put to test and Dennis even tested in the thread discussion SAR vs dmd -i. DMD -i was faster. Walter said that a cold run could be way faster though since the files won't need to be mem mapped.

Now ... to convince Walter that loading the whole archive into RAM once will be better than mem-mapping...

RAM is cheap and source code is not a big memory hit.

On 4/5/2024 5:20 PM, Adam Wilson wrote:
> Now ... to convince Walter that loading the whole archive into RAM once will be better than mem-mapping...
> 
> RAM is cheap and source code is not a big memory hit.

There's a switch in the source code to read it all at once or use memory mapping, so I could benchmark which is faster.

There's no significant difference, probably because the files weren't large enough.

BTW, I recall that executable files are not read into memory and then jumped to. They are memory-mapped files, this is so the executable can start up much faster. Pieces of the executable are loaded in on demand, although the OS will speculatively load in pieces, too.

On 4/4/2024 4:12 AM, Paolo Invernizzi wrote:
> My 2 cents: there will be NO advantages in compilation time.

Unfortunately, some things cannot be benchmarked until they are built.

On 10/04/2024 5:18 AM, Walter Bright wrote:
> BTW, I recall that executable files are not read into memory and then jumped to. They are memory-mapped files, this is so the executable can start up much faster. Pieces of the executable are loaded in on demand, although the OS will speculatively load in pieces, too.

That doesn't sound right.

Address randomization, Windows remapping of symbols at runtime (with state that is kept around so you can do it later), all suggest it isn't like that now.

On Tuesday, 9 April 2024 at 17:18:02 UTC, Walter Bright wrote:
> On 4/5/2024 5:20 PM, Adam Wilson wrote:
>> Now ... to convince Walter that loading the whole archive into RAM once will be better than mem-mapping...
>> 
>> RAM is cheap and source code is not a big memory hit.
>
> There's a switch in the source code to read it all at once or use memory mapping, so I could benchmark which is faster.
>
> There's no significant difference, probably because the files weren't large enough.
>
> BTW, I recall that executable files are not read into memory and then jumped to. They are memory-mapped files, this is so the executable can start up much faster. Pieces of the executable are loaded in on demand, although the OS will speculatively load in pieces, too.

Who managed to convince you to spend time working on this?

The solution to poor phobos build time is to stop abusing templates for code that doesn't exist for problems that nobody have

https://github.com/dlang/phobos/tree/master/phobos/sys

On 10/04/2024 5:22 AM, Richard (Rikki) Andrew Cattermole wrote:
> On 10/04/2024 5:18 AM, Walter Bright wrote:
>> BTW, I recall that executable files are not read into memory and then jumped to. They are memory-mapped files, this is so the executable can start up much faster. Pieces of the executable are loaded in on demand, although the OS will speculatively load in pieces, too.
> 
> That doesn't sound right.
> 
> Address randomization, Windows remapping of symbols at runtime (with state that is kept around so you can do it later), all suggest it isn't like that now.

It appears to have been true as of Windows 2000.

https://learn.microsoft.com/en-us/archive/msdn-magazine/2002/march/windows-2000-loader-what-goes-on-inside-windows-2000-solving-the-mysteries-of-the-loader

See: LdrpMapDll

However I don't think those two features may have existed at the time.

As of Windows Internals 5, the cache manager uses 256kb blocks as part of memory mapping (very useful information that!). Would be worth double checking that this is the default for std.mmap.

So it seems I'm half right, there is no way Windows could be memory mapping binaries when address randomization is turned on for a given block that has rewrites for symbol locations, but it may be memory mapping large blocks of data if it doesn't.

https://issues.dlang.org/show_bug.cgi?id=24494