On Wednesday, 10 April 2024 at 16:42:53 UTC, Steven Schveighoffer wrote:

C++ compilers are already on the next level, past PCH, with C++ modules.

VC++ uses a database format for BMI (Binary Module Interface), has open sourced it, and there are some people trying to champion it as means to have C++ tooling similar to what Java and .NET IDEs can do with JVM/CLR metadata.

https://devblogs.microsoft.com/cppblog/open-sourcing-ifc-sdk-for-cpp-modules/

On Thursday, 11 April 2024 at 01:36:57 UTC, Walter Bright wrote:

No. tar programs would work fine with it. We could indicate they are normal files, and normal tar programs would just extract an "index" file when expanding, or we could indicate they are vendor-specific extensions, which should be ignored or processed as normal files by other tar programs. We are not the first ones to think of these things, it is in the spec.

This would be a fallback, when an index isn't provided as the first file. So normal tar source files could be supported.

tar format doesn't have compression, though the tar executable supports it. I wouldn't recommend zip files as a supported archive format, and using compressed tarballs would definitely result in reading the whole file (you can't skip N bytes when you don't know the compressed size).

I stress again, we should not introduce esoteric formats that are mostly equivalent to existing formats without a good reason. The first option should be to use existing formats, seeing if we can fit our use case into them. If that is impossible or prevents certain features, then we can consider using a new format. It should be a high bar to add new file formats to the toolchain, as this affects all tools that people depend on and use.

Think of why we use standard object formats instead of our own format (which would allow much more tight integration with the language).

-Steve

On Thursday, 11 April 2024 at 15:28:34 UTC, Steven Schveighoffer wrote:

Sounds like a good solution. Users would be able to use e.g. any GUI program that supports tar to extract a file from the archive. The advantage is for reading. D-specific tools should be used to write the file. If there is any concern about this, it could even have a different extension so long as the file format is standard tar - users that know this can still benefit from tar readers. There seems to be precedent for this - apparently .jar files are .zip files.

Or just error if a tar file doesn't have the expected index file.

On Sunday, 14 April 2024 at 06:04:02 UTC, Walter Bright wrote:

Exactly, we don't need to be responsible for all the things. Using standard object format means we don't have to write our own linker.

Fair point. If this doesn't fly, then learning all the variations of tar might not be applicable (though I can say I personally "learned" tar in about 15 minutes, it's really simple).

This misses the point. It's not that it's easy to add to the compiler. Both are easy, both are straightforward, one might be easier than the other, but it's probably a wash (maybe 2 hours vs 4 hours?)

The problem is all the other tools that people might want to use. And specifically, I'm talking about IDEs. You have a 20 line solution in D, how does that help an IDE written in Java? However, Java has tar support that is tried and tested, and probably already in the IDE codebase itself.

Writing 20 lines of code isn't "mission accomplished". We now have to ask all IDE providers to support this for symbol lookup. That's what I'm talking about.

This is good, and I understand what you are trying to say. As long as it remains PoC, with the expectation that if it turns out to be useful, we address these ecosystem issues, then I have no objections.

No, vendor-specific sections are in the spec. Existing tar programs would still read these just fine.

But even if we wanted to avoid that, adding an index can be done by including a specific filename that the D compiler recognizes as the index.

Yes, of course. It's just, will there be a ready-made library available for whatever IDEs are using for language/libraries? With .sar, the answer is no (it hasn't been invented yet). With .tar, it's likely yes.

-Steve