Fossil

Check-in [b4beadb5]
Login

Check-in [b4beadb5]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Improvements to the way Fossil handles merging of private branches into public branches. The "closed" tag on the private branch is omitted so that it does not leak into the public branch causing a phantom. This is a start, but additional improvements are needed.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: b4beadb5078a03421a62f5a205cea1a52febe84b83c5189c9d271cd54a459136
User & Date: drh 2020-04-15 12:40:24
Persistent Phantoms Caused By Private Branches

We were having a problem where people would work on some feature in a private branch, then do a "fossil up trunk; fossil merge --integrate private;" to merge the private branch into trunk. The --integrate flag was causing a "closed" tag against the private branch to appear in the public block-chain. This generated phantoms (artifacts whose hash is known but whose content is unknown) which would then cause "gimme" cards to be sent in both directions on every sync round-trip.

One solution to these extra gimme cards is to make the phantoms as private. This was accomplished by the sync-improvements branch and subsequent fixes on trunk culminating with check-in b517a989d61772c2. In those patches, when the original repository that actually holds the private branch sees a "gimme" card with a private artifact, it replies with an "igot" card on the artifact with the private flag set. The sender of the "gimme" card then knows that the phantom is private, marks it as such in the "private" table of the database, and subsequent "gimme" cards are avoided.

There are problems with the "sync-improvements" approach:

  1. It relies on messages from the original holder of the private branch. If the repo that holds the private branch never again participates in a sync, the private "igot" card never gets sent, and we never learn that the artifact is private.

  2. The fact that the phantom artifact is private is stored in the "private" table of the database. This table persists across "fossil rebuild". But it is not part of the block-chain.

This check-in addresses the problem differently, by simply not creating the "closed" tag in the first place.

Related changes

Another series of edits on trunk culminating with check-in 941280ae0aa53d1d adds the /phantoms webpage and shows all public phantoms in the "Security Audit" page. This easily identifies phantoms and allows the administrator to manually create "private" tags against those phantoms in the block-chain, if desired.

Future enhancements

  1. This check-in prints a warning message if --integrate is used on a private branch merge, saying that the --integrate flag will be ignored and advising the user to run a separate "fossil amend" command. Better would be if the "fossil amend" was run automatically.

  2. On the /phantoms webpage and in the display of phantoms on Security Audit page, there ought to be a button or buttons that Admins can press that will automatically create "private" tags on those phantoms, so that the fact that the phantom is private becomes part of the blockchain.

  3. There ought to be a relatively simply mechanism to see the private phantoms generated by the "sync-improvement" changes and allow an Admin to insert "private" tags for those phantoms into the blockchain.

Private Merge Parents In P-Cards

Currently, when a private branch is merged into a public branch, the private branch is omitted from the P-card of the manifest. I think this should change. the P-card should reference the private merge parent. But, perhaps there should also be a separate (public) tag artifact that designates the private merge parent as being private, so that there will be an artifact in the public blockchain that prevents the P-card reference from generating a persistent phantom.

Backlinks to Markdown

This attached wiki page is formatted in Markdown. I am curious to see how the "backlink" handler deals with the references to check-ins and branches found in this note. There might be opportunities for improvement there. We'll see. Fixed by check-in ae1dac83d6bc4a3e.

Attach Forum Entries To Check-Ins and Branches?

This wiki pages seems like it might go better as a Forum page (albeit a Forum page in the actual source repository, not in the separate fossil-forum repo.) Should we extend the "associated wiki" mechanism so that it also allows "associated forum"?

Context
2020-04-16
22:35
Improvements to handling of backlinks: (1) Fossil now scans for backlinks in Markdown formatted ticket changes in addition to Fossil-Wiki formatted changes. (2) Wiki is scanned for backlinks, including wiki associated with check-ins and branches. (3) Much of the backlink code is gathered together into a single new "backlink.c" source file. (4) There are improvements to the timeline display of wiki changes. Ticket [a3572c6a5b47cd5a]. ... (check-in: ae1dac83 user: drh tags: trunk)
13:06
Begin breaking out the code for BACKLINK processing into a separate source file: backlink.c ... (check-in: 10c75204 user: drh tags: backlink-updates)
2020-04-15
12:40
Improvements to the way Fossil handles merging of private branches into public branches. The "closed" tag on the private branch is omitted so that it does not leak into the public branch causing a phantom. This is a start, but additional improvements are needed. ... (check-in: b4beadb5 user: drh tags: trunk)
11:28
Fix typo in the "css-tricks.md" document. ... (check-in: 9b0661aa user: drh tags: trunk)
11:12
Merge in trunk. ... (Closed-Leaf check-in: 7896afa7 user: florian tags: private-branches)
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to src/branch.c.

142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
  if( isPrivate && zColor==0 ) zColor = "#fec084";
  if( zColor!=0 ){
    blob_appendf(&branch, "T *bgcolor * %F\n", zColor);
  }
  blob_appendf(&branch, "T *branch * %F\n", zBranch);
  blob_appendf(&branch, "T *sym-%F *\n", zBranch);
  if( isPrivate ){
    blob_appendf(&branch, "T +private *\n");
    noSign = 1;
  }

  /* Cancel all other symbolic tags */
  db_prepare(&q,
      "SELECT tagname FROM tagxref, tag"
      " WHERE tagxref.rid=%d AND tagxref.tagid=tag.tagid"







<







142
143
144
145
146
147
148

149
150
151
152
153
154
155
  if( isPrivate && zColor==0 ) zColor = "#fec084";
  if( zColor!=0 ){
    blob_appendf(&branch, "T *bgcolor * %F\n", zColor);
  }
  blob_appendf(&branch, "T *branch * %F\n", zBranch);
  blob_appendf(&branch, "T *sym-%F *\n", zBranch);
  if( isPrivate ){

    noSign = 1;
  }

  /* Cancel all other symbolic tags */
  db_prepare(&q,
      "SELECT tagname FROM tagxref, tag"
      " WHERE tagxref.rid=%d AND tagxref.tagid=tag.tagid"

Changes to src/checkin.c.

1671
1672
1673
1674
1675
1676
1677








1678
1679
1680
1681
1682
1683
1684
                 " WHERE id %s ORDER BY 1",
                 p->integrateFlag ? "IN(0,-4)" : "=(-4)");
  while( db_step(&q)==SQLITE_ROW ){
    const char *zIntegrateUuid = db_column_text(&q, 0);
    int rid = db_column_int(&q, 1);
    if( is_a_leaf(rid) && !db_exists("SELECT 1 FROM tagxref "
        " WHERE tagid=%d AND rid=%d AND tagtype>0", TAG_CLOSED, rid)){








      blob_appendf(pOut, "T +closed %s\n", zIntegrateUuid);
    }
  }
  db_finalize(&q);

  if( p->azTag ){
    for(i=0; p->azTag[i]; i++){







>
>
>
>
>
>
>
>







1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
                 " WHERE id %s ORDER BY 1",
                 p->integrateFlag ? "IN(0,-4)" : "=(-4)");
  while( db_step(&q)==SQLITE_ROW ){
    const char *zIntegrateUuid = db_column_text(&q, 0);
    int rid = db_column_int(&q, 1);
    if( is_a_leaf(rid) && !db_exists("SELECT 1 FROM tagxref "
        " WHERE tagid=%d AND rid=%d AND tagtype>0", TAG_CLOSED, rid)){
#if 0
      /* Make sure the check-in manifest of the resulting merge child does not
      ** include a +close tag referring to the leaf check-in on a private
      ** branch, so as not to generate a missing artifact reference on
      ** repository clones without that private branch.  The merge command
      ** should have dropped the --integrate option, at this point. */
      assert( !content_is_private(rid) );
#endif
      blob_appendf(pOut, "T +closed %s\n", zIntegrateUuid);
    }
  }
  db_finalize(&q);

  if( p->azTag ){
    for(i=0; p->azTag[i]; i++){
2038
2039
2040
2041
2042
2043
2044


2045
2046
2047
2048
2049
2050
2051
  int nvid;              /* Blob-id of the new check-in */
  Blob comment;          /* Check-in comment */
  const char *zComment;  /* Check-in comment */
  Stmt q;                /* Various queries */
  char *zUuid;           /* UUID of the new check-in */
  int useHash = 0;       /* True to verify file status using hashing */
  int noSign = 0;        /* True to omit signing the manifest using GPG */


  int isAMerge = 0;      /* True if checking in a merge */
  int noWarningFlag = 0; /* True if skipping all warnings */
  int noPrompt = 0;      /* True if skipping all prompts */
  int forceFlag = 0;     /* Undocumented: Disables all checks */
  int forceDelta = 0;    /* Force a delta-manifest */
  int forceBaseline = 0; /* Force a baseline-manifest */
  int allowConflict = 0; /* Allow unresolve merge conflicts */







>
>







2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
  int nvid;              /* Blob-id of the new check-in */
  Blob comment;          /* Check-in comment */
  const char *zComment;  /* Check-in comment */
  Stmt q;                /* Various queries */
  char *zUuid;           /* UUID of the new check-in */
  int useHash = 0;       /* True to verify file status using hashing */
  int noSign = 0;        /* True to omit signing the manifest using GPG */
  int privateFlag = 0;   /* True if the --private option is present */
  int privateParent = 0; /* True if the parent check-in is private */
  int isAMerge = 0;      /* True if checking in a merge */
  int noWarningFlag = 0; /* True if skipping all warnings */
  int noPrompt = 0;      /* True if skipping all prompts */
  int forceFlag = 0;     /* Undocumented: Disables all checks */
  int forceDelta = 0;    /* Force a delta-manifest */
  int forceBaseline = 0; /* Force a baseline-manifest */
  int allowConflict = 0; /* Allow unresolve merge conflicts */
2075
2076
2077
2078
2079
2080
2081

2082
2083
2084
2085
2086
2087
2088
  int bRecheck = 0;      /* Repeat fork and closed-branch checks*/

  memset(&sCiInfo, 0, sizeof(sCiInfo));
  url_proxy_options();
  /* --sha1sum is an undocumented alias for --hash for backwards compatiblity */
  useHash = find_option("hash",0,0)!=0 || find_option("sha1sum",0,0)!=0;
  noSign = find_option("nosign",0,0)!=0;

  forceDelta = find_option("delta",0,0)!=0;
  forceBaseline = find_option("baseline",0,0)!=0;
  if( forceDelta && forceBaseline ){
    fossil_fatal("cannot use --delta and --baseline together");
  }
  dryRunFlag = find_option("dry-run","n",0)!=0;
  if( !dryRunFlag ){







>







2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
  int bRecheck = 0;      /* Repeat fork and closed-branch checks*/

  memset(&sCiInfo, 0, sizeof(sCiInfo));
  url_proxy_options();
  /* --sha1sum is an undocumented alias for --hash for backwards compatiblity */
  useHash = find_option("hash",0,0)!=0 || find_option("sha1sum",0,0)!=0;
  noSign = find_option("nosign",0,0)!=0;
  privateFlag = find_option("private",0,0)!=0;
  forceDelta = find_option("delta",0,0)!=0;
  forceBaseline = find_option("baseline",0,0)!=0;
  if( forceDelta && forceBaseline ){
    fossil_fatal("cannot use --delta and --baseline together");
  }
  dryRunFlag = find_option("dry-run","n",0)!=0;
  if( !dryRunFlag ){
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136

2137



2138






2139
2140
2141
2142
2143
2144
2145
    if( zTag[0]==0 ) continue;
    sCiInfo.azTag = fossil_realloc((void*)sCiInfo.azTag,
                                    sizeof(char*)*(nTag+2));
    sCiInfo.azTag[nTag++] = zTag;
    sCiInfo.azTag[nTag] = 0;
  }
  zComFile = find_option("message-file", "M", 1);
  if( find_option("private",0,0) ){
    g.markPrivate = 1;
    if( sCiInfo.zBranch==0 ) sCiInfo.zBranch = "private";
    if( sCiInfo.zBrClr==0 && sCiInfo.zColor==0 ){
      sCiInfo.zBrClr = "#fec084";  /* Orange */
    }
  }
  sCiInfo.zDateOvrd = find_option("date-override",0,1);
  sCiInfo.zUserOvrd = find_option("user-override",0,1);
  db_must_be_within_tree();
  noSign = db_get_boolean("omitsign", 0)|noSign;
  if( db_get_boolean("clearsign", 0)==0 ){ noSign = 1; }
  useCksum = db_get_boolean("repo-cksum", 1);
  outputManifest = db_get_manifest_setting();
  verify_all_options();

  /* Get the ID of the parent manifest artifact */
  vid = db_lget_int("checkout", 0);
  if( vid==0 ){
    useCksum = 1;
    if( sCiInfo.zBranch==0 ) {
      sCiInfo.zBranch=db_get("main-branch", 0);
    }

  }else if( content_is_private(vid) ){



    g.markPrivate = 1;






  }

  /* Do not allow the creation of a new branch using an existing open
  ** branch name unless the --force flag is used */
  if( sCiInfo.zBranch!=0
   && !forceFlag
   && fossil_strcmp(sCiInfo.zBranch,"private")!=0







<
<
<
<
<
<
<













|


>
|
>
>
>
|
>
>
>
>
>
>







2118
2119
2120
2121
2122
2123
2124







2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
    if( zTag[0]==0 ) continue;
    sCiInfo.azTag = fossil_realloc((void*)sCiInfo.azTag,
                                    sizeof(char*)*(nTag+2));
    sCiInfo.azTag[nTag++] = zTag;
    sCiInfo.azTag[nTag] = 0;
  }
  zComFile = find_option("message-file", "M", 1);







  sCiInfo.zDateOvrd = find_option("date-override",0,1);
  sCiInfo.zUserOvrd = find_option("user-override",0,1);
  db_must_be_within_tree();
  noSign = db_get_boolean("omitsign", 0)|noSign;
  if( db_get_boolean("clearsign", 0)==0 ){ noSign = 1; }
  useCksum = db_get_boolean("repo-cksum", 1);
  outputManifest = db_get_manifest_setting();
  verify_all_options();

  /* Get the ID of the parent manifest artifact */
  vid = db_lget_int("checkout", 0);
  if( vid==0 ){
    useCksum = 1;
    if( privateFlag==0 && sCiInfo.zBranch==0 ) {
      sCiInfo.zBranch=db_get("main-branch", 0);
    }
  }else{
    privateParent = content_is_private(vid);
  }

  /* Track the "private" status */
  g.markPrivate = privateFlag || privateParent;
  if( privateFlag && !privateParent ){
    /* Apply default branch name ("private") and color ("orange") if not
    ** specified otherwise on the command-line, and if the parent is not
    ** already private. */
    if( sCiInfo.zBranch==0 ) sCiInfo.zBranch = "private";
    if( sCiInfo.zBrClr==0 && sCiInfo.zColor==0 ) sCiInfo.zBrClr = "#fec084";
  }

  /* Do not allow the creation of a new branch using an existing open
  ** branch name unless the --force flag is used */
  if( sCiInfo.zBranch!=0
   && !forceFlag
   && fossil_strcmp(sCiInfo.zBranch,"private")!=0

Changes to src/json_branch.c.

264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
  if( content_is_private(rootid) ) zOpt->isPrivate = 1;
  if( zOpt->isPrivate && zColor==0 ) zColor = "#fec084";
  if( zColor!=0 ){
    blob_appendf(&branch, "T *bgcolor * %F\n", zColor);
  }
  blob_appendf(&branch, "T *branch * %F\n", zBranch);
  blob_appendf(&branch, "T *sym-%F *\n", zBranch);
  if( zOpt->isPrivate ){
    blob_appendf(&branch, "T +private *\n");
  }

  /* Cancel all other symbolic tags */
  db_prepare(&q,
      "SELECT tagname FROM tagxref, tag"
      " WHERE tagxref.rid=%d AND tagxref.tagid=tag.tagid"
      "   AND tagtype>0 AND tagname GLOB 'sym-*'"
      " ORDER BY tagname",
      rootid);
  while( db_step(&q)==SQLITE_ROW ){
    const char *zTag = db_column_text(&q, 0);
    blob_appendf(&branch, "T -%F *\n", zTag);
  }
  db_finalize(&q);

  blob_appendf(&branch, "U %F\n", g.zLogin);
  md5sum_blob(&branch, &mcksum);
  blob_appendf(&branch, "Z %b\n", &mcksum);

  brid = content_put(&branch);
  if( brid==0 ){
    fossil_panic("Problem committing manifest: %s", g.zErrMsg);
  }
  db_multi_exec("INSERT OR IGNORE INTO unsent VALUES(%d)", brid);
  if( manifest_crosslink(brid, &branch, MC_PERMIT_HOOKS)==0 ){
    fossil_panic("%s", g.zErrMsg);
  }







<
<
<


















|







264
265
266
267
268
269
270



271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
  if( content_is_private(rootid) ) zOpt->isPrivate = 1;
  if( zOpt->isPrivate && zColor==0 ) zColor = "#fec084";
  if( zColor!=0 ){
    blob_appendf(&branch, "T *bgcolor * %F\n", zColor);
  }
  blob_appendf(&branch, "T *branch * %F\n", zBranch);
  blob_appendf(&branch, "T *sym-%F *\n", zBranch);




  /* Cancel all other symbolic tags */
  db_prepare(&q,
      "SELECT tagname FROM tagxref, tag"
      " WHERE tagxref.rid=%d AND tagxref.tagid=tag.tagid"
      "   AND tagtype>0 AND tagname GLOB 'sym-*'"
      " ORDER BY tagname",
      rootid);
  while( db_step(&q)==SQLITE_ROW ){
    const char *zTag = db_column_text(&q, 0);
    blob_appendf(&branch, "T -%F *\n", zTag);
  }
  db_finalize(&q);

  blob_appendf(&branch, "U %F\n", g.zLogin);
  md5sum_blob(&branch, &mcksum);
  blob_appendf(&branch, "Z %b\n", &mcksum);

  brid = content_put_ex(&branch, 0, 0, 0, zOpt->isPrivate);
  if( brid==0 ){
    fossil_panic("Problem committing manifest: %s", g.zErrMsg);
  }
  db_multi_exec("INSERT OR IGNORE INTO unsent VALUES(%d)", brid);
  if( manifest_crosslink(brid, &branch, MC_PERMIT_HOOKS)==0 ){
    fossil_panic("%s", g.zErrMsg);
  }

Changes to src/merge.c.

403
404
405
406
407
408
409







410
411
412
413
414
415
416
    fossil_print("Merge skipped because it is a no-op. "
                 " Use --force to override.\n");
    return;
  }
  if( integrateFlag && !is_a_leaf(mid)){
    fossil_warning("ignoring --integrate: %s is not a leaf", g.argv[2]);
    integrateFlag = 0;







  }
  if( verboseFlag ){
    print_checkin_description(mid, 12,
              integrateFlag ? "integrate:" : "merge-from:");
    print_checkin_description(pid, 12, "baseline:");
  }
  vfile_check_signature(vid, CKSIG_ENOTFILE);







>
>
>
>
>
>
>







403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
    fossil_print("Merge skipped because it is a no-op. "
                 " Use --force to override.\n");
    return;
  }
  if( integrateFlag && !is_a_leaf(mid)){
    fossil_warning("ignoring --integrate: %s is not a leaf", g.argv[2]);
    integrateFlag = 0;
  }
  if( integrateFlag && content_is_private(mid) ){
    fossil_warning(
      "ignoring --integrate: %s is on a private branch"
      "\n Use \"fossil amend --close\" (after commit) to close the leaf.",
      g.argv[2]);
    integrateFlag = 0;
  }
  if( verboseFlag ){
    print_checkin_description(mid, 12,
              integrateFlag ? "integrate:" : "merge-from:");
    print_checkin_description(pid, 12, "baseline:");
  }
  vfile_check_signature(vid, CKSIG_ENOTFILE);

Changes to src/rebuild.c.

1138
1139
1140
1141
1142
1143
1144
1145
1146














































1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158


1159
1160
1161
1162
1163

1164

1165
1166
1167
1168
1169
1170
1171
    zFile,zHashPolicy);
  free(zFile);
  free(zFNameFormat);
  zFNameFormat = 0;
  cchFNamePrefix = 0;
}
#endif

/*














































** COMMAND: reconstruct*
**
** Usage: %fossil reconstruct ?OPTIONS? FILENAME DIRECTORY
**
** This command studies the artifacts (files) in DIRECTORY and
** reconstructs the fossil record from them. It places the new
** fossil repository in FILENAME. Subdirectories are read, files
** with leading '.' in the filename are ignored.
**
** Options:
**    -K|--keep-rid1    Read the filename of the artifact with
**                      RID=1 from the file .rid in DIRECTORY.


**
** See also: deconstruct, rebuild
*/
void reconstruct_cmd(void) {
  char *zPassword;

  fKeepRid1 = find_option("keep-rid1","K",0)!=0;

  if( g.argc!=4 ){
    usage("FILENAME DIRECTORY");
  }
  if( file_isdir(g.argv[3], ExtFILE)!=1 ){
    fossil_print("\"%s\" is not a directory\n\n", g.argv[3]);
    usage("FILENAME DIRECTORY");
  }









>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>




|
|
<
|


|
|
>
>





>

>







1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198

1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
    zFile,zHashPolicy);
  free(zFile);
  free(zFNameFormat);
  zFNameFormat = 0;
  cchFNamePrefix = 0;
}
#endif

/*
** Helper functions used by the `deconstruct' and `reconstruct' commands to
** save and restore the contents of the PRIVATE table.
*/
void private_export(char *zFileName)
{
  Stmt q;
  Blob fctx = empty_blob;
  blob_append(&fctx, "# The UUIDs of private artifacts\n", -1);
  db_prepare(&q,
    "SELECT uuid FROM blob WHERE rid IN ( SELECT rid FROM private );");
  while( db_step(&q)==SQLITE_ROW ){
    const char *zUuid = db_column_text(&q, 0);
    blob_append(&fctx, zUuid, -1);
    blob_append(&fctx, "\n", -1);
  }
  db_finalize(&q);
  blob_write_to_file(&fctx, zFileName);
  blob_reset(&fctx);
}
void private_import(char *zFileName)
{
  Blob fctx;
  if( blob_read_from_file(&fctx, zFileName, ExtFILE)!=-1 ){
    Blob line, value;
    while( blob_line(&fctx, &line)>0 ){
      char *zUuid;
      int nUuid;
      if( blob_token(&line, &value)==0 ) continue;  /* Empty line */
      if( blob_buffer(&value)[0]=='#' ) continue;   /* Comment */
      blob_trim(&value);
      zUuid = blob_buffer(&value);
      nUuid = blob_size(&value);
      zUuid[nUuid] = 0;
      if( hname_validate(zUuid, nUuid)!=HNAME_ERROR ){
        canonical16(zUuid, nUuid);
        db_multi_exec(
          "INSERT OR IGNORE INTO private"
          " SELECT rid FROM blob WHERE uuid = %Q;",
          zUuid);
      }
    }
    blob_reset(&fctx);
  }
}

/*
** COMMAND: reconstruct*
**
** Usage: %fossil reconstruct ?OPTIONS? FILENAME DIRECTORY
**
** This command studies the artifacts (files) in DIRECTORY and reconstructs the
** Fossil record from them.  It places the new Fossil repository in FILENAME.

** Subdirectories are read, files with leading '.' in the filename are ignored.
**
** Options:
**   -K|--keep-rid1     Read the filename of the artifact with RID=1 from the
**                      file .rid in DIRECTORY.
**   -P|--keep-private  Mark the artifacts listed in the file .private in
**                      DIRECTORY as private in the new Fossil repository.
**
** See also: deconstruct, rebuild
*/
void reconstruct_cmd(void) {
  char *zPassword;
  int fKeepPrivate;
  fKeepRid1 = find_option("keep-rid1","K",0)!=0;
  fKeepPrivate = find_option("keep-private","P",0)!=0;
  if( g.argc!=4 ){
    usage("FILENAME DIRECTORY");
  }
  if( file_isdir(g.argv[3], ExtFILE)!=1 ){
    fossil_print("\"%s\" is not a directory\n\n", g.argv[3]);
    usage("FILENAME DIRECTORY");
  }
1180
1181
1182
1183
1184
1185
1186


1187






1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221



1222
1223
1224
1225
1226
1227
1228

1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244


1245
1246
1247
1248
1249
1250
1251
  db_initial_setup(0, 0, 0);

  fossil_print("Reading files from directory \"%s\"...\n", g.argv[3]);
  recon_read_dir(g.argv[3]);
  fossil_print("\nBuilding the Fossil repository...\n");

  rebuild_db(0, 1, 1);


  reconstruct_private_table();







  /* Skip the verify_before_commit() step on a reconstruct.  Most artifacts
  ** will have been changed and verification therefore takes a really, really
  ** long time.
  */
  verify_cancel();

  db_end_transaction(0);
  fossil_print("project-id: %s\n", db_get("project-code", 0));
  fossil_print("server-id: %s\n", db_get("server-code", 0));
  zPassword = db_text(0, "SELECT pw FROM user WHERE login=%Q", g.zLogin);
  fossil_print("admin-user: %s (initial password is \"%s\")\n", g.zLogin, zPassword);
}

/*
** COMMAND: deconstruct*
**
** Usage %fossil deconstruct ?OPTIONS? DESTINATION
**
**
** This command exports all artifacts of a given repository and
** writes all artifacts to the file system. The DESTINATION directory
** will be populated with subdirectories AA and files AA/BBBBBBBBB.., where
** AABBBBBBBBB.. is the 40+ character artifact ID, AA the first 2 characters.
** If -L|--prefixlength is given, the length (default 2) of the directory
** prefix can be set to 0,1,..,9 characters.
**
** Options:
**   -R|--repository REPOSITORY  Deconstruct given REPOSITORY.
**   -K|--keep-rid1              Save the filename of the artifact with RID=1 to
**                               the file .rid1 in the DESTINATION directory.
**   -L|--prefixlength N         Set the length of the names of the DESTINATION
**                               subdirectories to N.
**   --private                   Include private artifacts.



**
** See also: rebuild, reconstruct
*/
void deconstruct_cmd(void){
  const char *zPrefixOpt;
  Stmt        s;
  int privateFlag;


  fKeepRid1 = find_option("keep-rid1","K",0)!=0;
  /* get and check prefix length argument and build format string */
  zPrefixOpt=find_option("prefixlength","L",1);
  if( !zPrefixOpt ){
    prefixLength = 2;
  }else{
    if( zPrefixOpt[0]>='0' && zPrefixOpt[0]<='9' && !zPrefixOpt[1] ){
      prefixLength = (int)(*zPrefixOpt-'0');
    }else{
      fossil_fatal("N(%s) is not a valid prefix length!",zPrefixOpt);
    }
  }
  /* open repository and open query for all artifacts */
  db_find_and_open_repository(OPEN_ANY_SCHEMA, 0);
  privateFlag = find_option("private",0,0)!=0;


  verify_all_options();
  /* check number of arguments */
  if( g.argc!=3 ){
    usage ("?OPTIONS? DESTINATION");
  }
  /* get and check argument destination directory */
  zDestDir = g.argv[g.argc-1];







>
>

>
>
>
>
>
>



















<
|
|
|
|
|
|








>
>
>

|





>
















>
>







1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263

1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
  db_initial_setup(0, 0, 0);

  fossil_print("Reading files from directory \"%s\"...\n", g.argv[3]);
  recon_read_dir(g.argv[3]);
  fossil_print("\nBuilding the Fossil repository...\n");

  rebuild_db(0, 1, 1);

  /* Backwards compatibility: Mark check-ins with "+private" tags as private. */
  reconstruct_private_table();
  /* Newer method: Import the list of private artifacts to the PRIVATE table. */
  if( fKeepPrivate ){
    char *zFnDotPrivate = mprintf("%s/.private", g.argv[3]);
    private_import(zFnDotPrivate);
    free(zFnDotPrivate);
  }

  /* Skip the verify_before_commit() step on a reconstruct.  Most artifacts
  ** will have been changed and verification therefore takes a really, really
  ** long time.
  */
  verify_cancel();

  db_end_transaction(0);
  fossil_print("project-id: %s\n", db_get("project-code", 0));
  fossil_print("server-id: %s\n", db_get("server-code", 0));
  zPassword = db_text(0, "SELECT pw FROM user WHERE login=%Q", g.zLogin);
  fossil_print("admin-user: %s (initial password is \"%s\")\n", g.zLogin, zPassword);
}

/*
** COMMAND: deconstruct*
**
** Usage %fossil deconstruct ?OPTIONS? DESTINATION
**

** This command exports all artifacts of a given repository and writes all
** artifacts to the file system.  The DESTINATION directory will be populated
** with subdirectories AA and files AA/BBBBBBBBB.., where AABBBBBBBBB.. is the
** 40+ character artifact ID, AA the first 2 characters.
** If -L|--prefixlength is given, the length (default 2) of the directory prefix
** can be set to 0,1,..,9 characters.
**
** Options:
**   -R|--repository REPOSITORY  Deconstruct given REPOSITORY.
**   -K|--keep-rid1              Save the filename of the artifact with RID=1 to
**                               the file .rid1 in the DESTINATION directory.
**   -L|--prefixlength N         Set the length of the names of the DESTINATION
**                               subdirectories to N.
**   --private                   Include private artifacts.
**   -P|--keep-private           Save the list of private artifacts to the file
**                               .private in the DESTINATION directory (implies
**                               the --private option).
**
** See also: reconstruct, rebuild
*/
void deconstruct_cmd(void){
  const char *zPrefixOpt;
  Stmt        s;
  int privateFlag;
  int fKeepPrivate;

  fKeepRid1 = find_option("keep-rid1","K",0)!=0;
  /* get and check prefix length argument and build format string */
  zPrefixOpt=find_option("prefixlength","L",1);
  if( !zPrefixOpt ){
    prefixLength = 2;
  }else{
    if( zPrefixOpt[0]>='0' && zPrefixOpt[0]<='9' && !zPrefixOpt[1] ){
      prefixLength = (int)(*zPrefixOpt-'0');
    }else{
      fossil_fatal("N(%s) is not a valid prefix length!",zPrefixOpt);
    }
  }
  /* open repository and open query for all artifacts */
  db_find_and_open_repository(OPEN_ANY_SCHEMA, 0);
  privateFlag = find_option("private",0,0)!=0;
  fKeepPrivate = find_option("keep-private","P",0)!=0;
  if( fKeepPrivate ) privateFlag = 1;
  verify_all_options();
  /* check number of arguments */
  if( g.argc!=3 ){
    usage ("?OPTIONS? DESTINATION");
  }
  /* get and check argument destination directory */
  zDestDir = g.argv[g.argc-1];
1305
1306
1307
1308
1309
1310
1311








1312
1313
1314
1315
1316
1317
1318
1319
        Blob content;
        content_get(rid, &content);
        rebuild_step(rid, size, &content);
      }
    }
  }
  db_finalize(&s);








  if(!g.fQuiet && ttyOutput ){
    fossil_print("\n");
  }

  /* free filename format string */
  free(zFNameFormat);
  zFNameFormat = 0;
}







>
>
>
>
>
>
>
>








1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
        Blob content;
        content_get(rid, &content);
        rebuild_step(rid, size, &content);
      }
    }
  }
  db_finalize(&s);

  /* Export the list of private artifacts. */
  if( fKeepPrivate ){
    char *zFnDotPrivate = mprintf("%s/.private", zDestDir);
    private_export(zFnDotPrivate);
    free(zFnDotPrivate);
  }

  if(!g.fQuiet && ttyOutput ){
    fossil_print("\n");
  }

  /* free filename format string */
  free(zFNameFormat);
  zFNameFormat = 0;
}

Changes to www/private.wiki.

29
30
31
32
33
34
35
36
37
38
39














40
41
42
43
44
45
46

<blockquote><pre>
fossil update trunk
fossil merge private
fossil commit
</pre></blockquote>

The private branch remains private.  (There is no way to convert a private
branch into a public branch.)  But all of the changes associated with
the private branch are now folded into the public branch and are hence
visible to other users of the project.















<h2>Syncing Private Branches</h2>

A private branch normally stays on the one repository where it was
originally created.  But sometimes you want to share private branches
with another repository.  For example, you might be building a cross-platform
application and have separate repositories on your Windows laptop,







<
|


>
>
>
>
>
>
>
>
>
>
>
>
>
>







29
30
31
32
33
34
35

36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

<blockquote><pre>
fossil update trunk
fossil merge private
fossil commit
</pre></blockquote>


The private branch remains private, but all of the changes associated with
the private branch are now folded into the public branch and are hence
visible to other users of the project.

A private branch created with Fossil version 1.30 or newer can also be
converted into a public branch using the <code>fossil publish</code>
command.  However, there is no way to convert a private branch created with
older versions of Fossil into a public branch.

The <code>--integrate</code> option of <code>fossil merge</code> (to close
the merged branch when committing) is ignored for a private branch -- or the
check-in manifest of the resulting merge child would include a
<code>+close</code> tag referring to the leaf check-in on the private branch,
and generate a missing artifact reference on repository clones without that
private branch.  It's still possible to close the leaf of the private branch
(after committing the merge child) with the <code>fossil amend --close</code>
command.

<h2>Syncing Private Branches</h2>

A private branch normally stays on the one repository where it was
originally created.  But sometimes you want to share private branches
with another repository.  For example, you might be building a cross-platform
application and have separate repositories on your Windows laptop,