I'm in the 'Timeline' section browser window of the UI clicking on a "Leaf check-in:" hash and the browser displays:

This page isn’t working

localhost didn’t send any data.

ERR_EMPTY_RESPONSE

In the console window where I started the Fossil UI server, a few of these messages appear (I've renamed the 'fossil' executable to 'fsl' for easier typing):

fsl: ./src/diff.c:2481: longestCommonSequence: Assertion `i>=iSXb && i>=iSXp' failed.

fsl: ./src/diff.c:2481: longestCommonSequence: Assertion `i>=iSXb && i>=iSXp' failed.

fsl: ./src/diff.c:2481: longestCommonSequence: Assertion `i>=iSXb && i>=iSXp' failed.

fsl: ./src/diff.c:2481: longestCommonSequence: Assertion `i>=iSXb && i>=iSXp' failed.

Clicking on other hash values in the Timeline window seems to work fine. Everything has been working well for a few days until I encountered this issue.

I'm new to using Fossil. I love all the Fossil concepts as it seems to provide a much more natural and intuitive workflow for me. I'm also really happy to get rid of using terse, hard-to-remember, unnatural Git commands locally (officially, Git is still our corporate workflow). I hope I can get these last few issues worked out and use Fossil 99% of the time and only touch Git when I'm checking into our corporate repo. Our policy is to have 'clean' Git commits only, so keeping all of the Fossil detail of "how things actually happened" in my private Fossil repo works well for me.

Thanks for all the work on Fossil and Tcl!

Ray

That assert() is in the logic that computes a "diff". You may have discovered a bug. Can you share the repo that you are using that gives this result so that we can repro the problem and then fix it?

Hi Richard,

I'm sorry, the repo contains sensitive corporate data. I'm not allowed to share it. Are is there other information I can provide that might help your debugging effort?

Ray

Are is there other information I can provide that might help your debugging effort?

Perhaps you can use:

$ fossil xdiff ...

where ... are local files which refer to the content the diff is crashing on. If those particular files aren't sensitive, then they would be helpful in diagnosing the problem. The code in question is nearly 3 years old and this is the first such account, so there's something odd about your data which would be very interesting to see.

That said: if we could get a copy of the repo, it would make diagnosing the problem much easier. If it's any consolation, Richard deals daily with proprietary data from corporate backers of the SQLite project and has a proven track record of not leaking any of that information to others.

I really want to help the debugging effort, but my hands are tied. I understand how frustrating it can be not being able to reproduce the sequence of events that lead to the error message. I'm under corporate and government restrictions prohibiting me from sharing the data.

The error message doesn't indicate which files Fossil/diff was working on, so I don't think I have enough information to effective run the 'fossill xdiff' test case.

This is fossil version 2.23 [47362306a7] 2023-11-01 18:56:47 UTC

My project isn't a typical software one. My build artifacts are large and take several hours to build. As a result, I'm storing a few of the build artifacts in the Fossil repo to allow me to rapidly switch branches and have easy access to the source as well as some of the build artifacts. I know storing binary build products in a repo is typically discouraged, but I've been trying to find a reliable way of being able to switch branches and avoid a several hour delay rebuilding everything to get back to "where I was" prior to the latest branch switch. As a result, the Fossil SQLite database file that encompasses the repo is about 256,000,000 bytes.

If this is a crazy use-case for Fossil, please let me know.

I really want to help the debugging effort, but my hands are tied.

Then you must understand that our hands are tied as well :/. We can't debug what we can't reproduce, and so far we've had precisely one report of this behavior in the past 17+ years.

As a result, the Fossil SQLite database file that encompasses the repo is about 256,000,000 bytes.

That size is not a problem. The current TCL core repository is, on my machine, 345mb (that size can vary depending on how deltas get applied on any given copy). Huge blobs (somewhere approaching 2gb) can't find their way into the repository - fossil will die loudly before that can happen. It is, however, hypothetically possible that you've uncovered a case where the blobs are within legal ranges but the memory required to render the diff is on some uncomfortable border (likely just shy of 2gb) where something is misbehaving. That's all speculation, though.

I know storing binary build products in a repo is typically discouraged...

The assertion you're getting to should not be triggerable by binary files: they hypothetically can't make it to that point in the diff process. Visual diffs won't be generated for any content which fossil believes to be "binary," noting that the distinction between "binary" and "text" requires heuristics-driven decision-making and it is certainly possible that you've hit a case where those heuristics fail. (Even so, that "shouldn't" cause that given assertion to trigger.)

If this is a crazy use-case for Fossil, please let me know.

We don't know because we can't see it ;).

It's good to know that Fossil by-design should be able to handle my use case. The blobs are binary, and not text files, so if there is a filter to prevent their diff, it's probably recognizing them correctly. Thanks to Stephan and Richard for responding so quickly. I know you can't spend tons of time chasing down a hard to reproduce corner case, so I'll see if I can find a good way to work around the issue. If not, I'll likely give Mercurial another try. I used it at another job before I knew Fossil even existed.

Fossil's forum, wiki, tickets are especially helpful when I have several development branches active. I use the documentation features to try to maintain sanity by keeping notes of my observations of each build. Maybe others have had similar ideas, but it seems like it would be nice to have CLI interface access to the the documentation features. Something like:

 fossil forum add -m "This build is working for xxxxx, but is broken doing this xxxxx function"

 fossil ticket add -m "xxxxx function broken in hash ###### build.  TODO fix the routine that is causing xxxxx function to break."

 fossil ticket list

Obviously formatting and the graphical features would be missing, but I wouldn't have to have a Fossil server constantly running to allow access.

Mercurial doesn't have any of the built-in documentation features, so the great way of keeping track of the state of each build in Fossil will have to degrade back to a "notes" text file embedded within each branch with Mercurial.

Stephan:

The assertion you're getting to should not be triggerable by binary files: they hypothetically can't make it to that point in the diff process.

anonymous:

The blobs are binary, and not text files, so if there is a filter to prevent their diff, it's probably recognizing them correctly.

Hm, no, obviously not ...

If "the blobs are binary" means that all the blobs involved in the diff hitting the assertion failure are binary, then they obviously were not filtered out of the diffs. In this case, we need a fix for the heuristics to detect binary files, and not for the diff algorithm. Is that correct?

In this case, we need a fix for the heuristics to detect binary files, and not for the diff algorithm. Is that correct?

That's my current hypothesis, but it's based on so little info that my confidence in it is not especially high.

Can you turn on fossil logging and/or debug features so that you might capture what is going on? Or compile a debug version to test with?

Can you turn on fossil logging and/or debug features ...

We don't have a level of logging which would tell us which artifacts are about to be diffed. However, attaching a debugger and capturing the stack trace at the assert() might be helpful even if we can't see the offending blobs.

Sorry I would off work last Friday the 25th and have been buried with work today. I'll try the bisect function when time allows.

Andy,

I think I've done roughly what you asked, but I didn't use the bisect command. I opened the "Timeline" view, checked the "Files" checkbox and clicked on the "[diff]" option for each of the files in the commit that produces the error.

All of the files are able to be diff'ed except for one. The offender is a 55 MByte file in the format of Intel Hex (i.e. a *.mcs type file - which is a text file and not binary - https://en.wikipedia.org/wiki/Intel_HEX ). There are two of these .mcs programming files produced during the build/compilation process. The other .mcs file (same size) is able to be diff'ed correctly. There is also a 40 MB build-artifact binary-file that I also store, but diff'ing it seem to work correctly as the message "cannot compute difference between binary files" is displayed.

If you can embed some debug code into the routines that cause the fault assertion and produce a special Fossil debug build (just for me to obtain information about this fault). I'll be happy to run it and provide as much debugging information as my restrictions permit me to.

Thanks,

Ray

All of the files are able to be diff'ed except for one. The offender is a 55 MByte file in the format of Intel Hex

That is enough information for us to improve the heuristics. diff will currently refuse to diff anything with more than a certain number of diff lines (10k, IIRC).

If there's no general objection to it, i'll add a heuristic of "don't bother trying to diff anything obscenely large," (say, 20mb(?), noting that sqlite3.c is 9.2mb). Would that be a fundamentally wrong thing to do?

10K seems a bit small. Doesn't matter if this kind of ascii hex file is big or small, the diff will never be useful.

Those ascii hex file are not like a source file (like sqlite3.c), if a few bytes is added, all the file will be shifted and 100% of the line will become different. (including the address on the 1st column). No common block will be found, but fossil will try to match some random unrelated sequence of ascii hex code.

I guess we can have global hardcoded limit, but I believe it should be bigger. It's hard to pick an exact value for this kind of limit, especially when hard coded.

10K seems a bit small. Doesn't matter if this kind of ascii hex file is big or small, the diff will never be useful.

To clarify: that's the number of diff lines, not the input file size. As you say, such large diffs aren't useful to humans.

The "too big?" heuristic is conditional, based on a flag passed to the diff calculator:

#define DIFF_TOO_MANY_CHANGES \
    "more than 10,000 changes\n"
...
  if( (pCfg->diffFlags & DIFF_NOTTOOBIG)!=0 ){
    int i, m, n;
    int *a = c.aEdit;
    int mx = c.nEdit;
    for(i=m=n=0; i<mx; i+=3){ m += a[i]; n += a[i+1]+a[i+2]; }
    if( n>10000 ){
      fossil_free(c.aFrom);
      fossil_free(c.aTo);
      fossil_free(c.aEdit);
      if( pOut ) diff_errmsg(pOut, DIFF_TOO_MANY_CHANGES, pCfg->diffFlags);
      return 0;
    }
  }

That condition has (in some form) been used in the web-rendering diff bits since the times of the cavepeople. It is not used in the CLI-mode diff (diffcmd.c), even when producing HTML output.

To clarify: that's the number of diff lines, not the input file size. As you say, such large diffs aren't useful to humans.

I understood that it's lines. But if I understand the piece of code above and it's really: "Number of CHANGED lines".. then it's a different story and yes, it make sense.

I tough it was representing the total number of lines in the file, I think in that case it would not be enough.

IMO, Looking at the diff of sqlite3.c file can be useful to human.. (if it doesn't have >10K of changed lines)

what's wrong with diffing large text files in general?

My suggestion about a size limit was before Martin reminded us about binary-glob, which i think is the OP's best choice all-around.

Even so, it would be really interesting to find the reason this is triggering an assert.

@OP: would it be possible anonymize the offending blob (simply replace all of the "data" parts with random ASCII data or some such) and provide us a copy of that? Perhaps that would be sufficient for us to reproduce the problem.

As a work around, probably you could add *.mcs to the binary-glob setting, so fossil will stop trying to diff or merge those file..

Anyway, I don't think trying to merge those kind of ascii hex file make sense..

Two things are making my whiskers twitch.

First is a memory of once having to work to a design rule that said limits should be at least 20 times the current known requirement. I will try to ignore that one...

Second is the mention in the spec that the line terminator characters are optional. Perhaps the issue is that diff is hitting really long lines and would struggle just as much with 20mb as 50. Is there a trap for long lines as well as 10k(?) lines in a difference?

Perhaps the issue is that diff is hitting really long lines and would struggle just as much with 20mb as 50.

Fossil invariably treats files with "really long lines" as binary (meaning only that it won't attempt to create "diffs" for them - it can/does still internally create deltas for them). That number is internally configurable/variable but is limited to 15 bits (32k).

IIRC, but am not certain of this and am not readily seeing it in human-readable code form, the diff defaults to 1024 for the line-length limit.

My .mcs file is almost 1,220,000 lines long with a line width of 42 characters. The last two lines are shorter than the rest. Following the Intel Hex format, the first character of each line is a colon ':' .

As I mentioned before, there are two such files in the repo. One doesn't appear to cause any problems, but the other does. They both appear to be the same size, but have differing content.

My .mcs file is almost 1,220,000 lines long ...

Martin's suggestion of using the binary-glob is most definitely the quick fix (and probably sensible for those files, anyway).

Note that fossil's only special handling for "binary" is that such files are not diff'd for purposes of the "diff" command, /vdiff page, etc. Fossil still internally creates and uses "deltas" (a lower-level form of diff) for binary files, so they still benefit from that level of compression. As the distinction between "binary" and "text" files is a human one, not a technical one, fossil notion of binary is essentially "this file is not suitable for display as-is to humans."

Hi Martin,

Merging a .mcs file doesn't make sense, but diff'ing them is sometimes useful. In cases where only a small change is made to the source file, it's reassuring to see the equivalent small change in a bit or a few bytes occur in the output build artifact. But yes, if it fixes the tool and prevents the fault, I'm all for the change.

I added a space-delimited "*.mcs" to the binary-glob setting, killed and restarted the fossil server and I still see the fault on the Fossil webpage. I already had an entry in binary-glob to ignore the 40 MByte binary I had stored.

All of the files are able to be diff'ed except for one. The offender is a 55 MByte file in the format of Intel Hex

Can you please try running the diff from the command line and see if it barfs in the same way (my expectation is that it will):

$ fossil diff  -y --webpage --from SOME_VERSION --to VERSION_WHICH_BREAKS > /dev/null

We can probably get away with "too beaucoup for a web page" output but probably not from the CLI (in that folks can reasonably expect a diff of any fossil-supported size to work from the CLI).

Yes, it looks like the same fault:

$ fossil diff -y --webpage --from 06d2aa --to 992c4cc > /dev/null

fossil: ./src/diff.c:2481: longestCommonSequence: Assertion `i>=iSXb && i>=iSXp' failed. Aborted (core dumped)

Ray - can you send to me and me only the two 55MB .mcs files? I can execute an NDA, if that helps. I can provide you with an FTP location to which you can upload them.

Hi Richard - Thank you very much for your offer to help. I'm sorry I can't provide the files. There are government restrictions covering this material.

If you could possibly create a debug build of Fossil, I can probably run it on one of our machines and provide you the results. Since it would be a non-standard tool, we would probably run it on an air-gapped computer and manually type the debug information back on a networked machine. I would have to gain approval to even do that.

This is causing much more work and trouble than I had intended. Fossil is a great tool and I intend to extensively use it for my personal work. I know it's important to find bugs in the software you create. I think it's great that there is such enthusiasm to make to tool even better, but I'm highly restricted on the ability to provide materials for you to use to debug.

I'll proceed as far as I can on the repo that causes the fault. When I get to a stable point in development, I might just try to manually recreate everything to see if it's some odd step I performed that caused the problem. Maybe exporting to Git then re-importing back to Fossil would make things easier and more accurate.

Thanks for all your help! However, I hate this feeling of getting so many people involved, but not being able to provide much help. I'll occasionally check this thread, but I'm trying to solve a few problems with my design and need to focus at this time, so I won't check it very often after tonight. Please don't spend an excess of time chasing this - how many people want to store huge build artifacts in a repo anyway? Thanks!

...converted it to Intel Hex using bin2hex.py...

Outstanding!

Because libfossil's diff code is very nearly a 1-to-1 port of fossil's i had to try this out there and, sure enough:

[stephan@nuc:~/tmp/intelhex]$ f-vdiff --from prev
f-vdiff: src/diff.c:188: fsl__diff_lcs: Assertion `i>=iSXb && i>=iSXp' failed.
Aborted (core dumped)

Edit: in the off chance that this might give Richard a clue:

(gdb) p i
$1 = 332904
(gdb) p iSXb
$2 = 111435
(gdb) p iSXp
$3 = 332909

Those numbers are identical in the crashes from both fossil and libfossil.

Somebody please send me the two files. Persumably running "fossil xdiff FILE1 FILE2" should also produce the error, no?

You can clone the Andy test repo ( see this post )

Problem will happens with fossil ui if you click on the last checkins /info page.

The problem also happens with "xdiff" on the 2 latest version of the file.

• Something interesting, doing the diff between the 2 same file with "--invert" will not trigger the problem.

Thank you. Please try again with the latest check-in.

Hi Richard,

I was able to fix my project problem and had a few minutes to read the forum messages. I downloaded Fossil version 2.25r1 and I can confirm your fix works on my original repo. I repeated all the actions that previously resulted in an error and now Fossil appears to work perfectly! Thank you and thanks to the team for working so quickly to find and solve the issue.

I'm really impressed with Fossil and the people behind it. Please keep up the great work!

Thank you very much!

Ray

in my terminal I see the following

Now, that one bug has just begun walking down its green mile and all this is becoming utterly uninteresting... (yawns). What really, really keeps this thread alive is the content of those insidious Intel hex files... 😂

(joke over)