)]}'
{
  "commit": "5ca252314bf097bfec299ea8564edc4885c1bd6f",
  "tree": "2074c563abca367e33eeb4d5029988d7c5d17392",
  "parents": [
    "230b58158e2bee20d02d4c94ecdbf87c4afd8c21"
  ],
  "author": {
    "name": "Lewis Hyatt",
    "email": "lhyatt@gmail.com",
    "time": "Sat Jul 11 14:21:12 2026 -0400"
  },
  "committer": {
    "name": "Lewis Hyatt",
    "email": "lhyatt@gcc.gnu.org",
    "time": "Fri Jul 17 08:46:45 2026 -0400"
  },
  "message": "lto: Overhaul approach to location streaming [PR65536]\n\nPR65536 is about location-related issues that arise when the LTO front end\nreads one or several large object files and runs out of location_t space to\nencode all the locations. The main reason for the potential problem is that\nthe libcpp linemap is designed to be used for incrementally reading source\nfiles (and their included header files) in the natural order. When it is\nused in another way, such as by LTO, which adds locations in the order in\nwhich it happened to read different entities, then the assumptions that\njustify its design are no longer applicable, and it is not hard to run out\nof location_t values (e.g., because of a large number of file changes, or\nlines being added out of order.)\n\nThe PR remains open because it is not theoretically resolved, but it has\nbeen resolved for all practical purposes by the following two improvements:\n\n    1) The lto_location_cache class now tries hard to optimize the number of\n       maps that it creates, especially by sorting the locations before\n       adding them. This approach only goes so far, because it can only work\n       at the LTO section level, so each function is handled independently\n       of the others, but it helps a lot with reducing the number of maps\n       required for a single function.\n\n    2) We moved to 64-bit location_t, which means everything is a lot more\n       forgiving of wasting location_t space.\n\nPoint 2) made this not much of an issue in practice, although sufficiently\nlarge files (especially with very long lines) could still trigger a problem\nin theory. While locations are now by and large working fine in LTO, there\nwas an interesting discussion on PR65536, starting around:\n\n    https://gcc.gnu.org/bugzilla/show_bug.cgi?id\u003d65536#c8\n\nabout how the location streaming might be optimized by streaming the linemap\nstructure itself, rather than a (file name, line number, column number)\ntriplet for each location. In the end, this approach was not taken because\nsolution 1) above was adequate and less disruptive. But the arguments\npresented in favor of streaming the linemap directly are still interesting.\n\nThat discussion did not touch on another topic, namely the need to make\n`#pragma GCC diagnostic\u0027 usable in LTO. Right now, it does not work, because\nthe diagnostic pragmas are not streamed out. But even if they were streamed\nout, the existing approach to locations in the LTO front end is not\ncompatible with enforcing the pragmas. Diagnostic pragmas require a global\nordering on location_t values, so that the machinery in option-classifier.cc\ncan determine which pragmas are in force at each location. This requires\nremembering the order in which each source line was encountered, and that is\nexactly what is encoded in the linemap. This cannot be reconstructed just\nfrom source file names and line numbers; consider, for example, that the\nsame file might be used in different translation units or multiple times in\nthe same translation unit with different diagnostic pragmas in place each\ntime.\n\nThese considerations tip the scales in favor of adding linemap streaming as\nManuel proposed on the PR. This patch prepares to support `#pragma GCC\ndiagnostic\u0027 in the LTO front end by changing the approach to location\nstreaming along these lines. The new approach has these general\ncharacteristics:\n\n    o There is a new LTO section (LTO_section_linemap) that contains the\n      information needed to reconstruct the linemap. There is an entry for\n      each line_map_ordinary object that was used by at least one\n      streamed-out location. When the LTO front end reads one of these map\n      entries, it adds a new map with the corresponding properties to its\n      own linemap using the new line_map_add_raw_map interface in libcpp.\n\n    o When a location needs to be output, we stream out two integers: one to\n      identify which linemap contains the location, and one containing the\n      offset from the start of that map to the location.\n\n    o When the language front ends stream out their data, they produce a\n      single linemap section (labeled as linemap.0) that applies to the\n      whole translation unit. When WPA prepares partitioned files for\n      LTRANS, it may copy function bodies into the LTRANS files without\n      reading them, so it needs also to copy the linemap sections they refer\n      to. Since the same linemap section will often be needed by more than\n      one partition, this is done by putting all of the linemap sections\n      into one additional object file, which is provided as input to each\n      LTRANS process via the new option -fltrans-linemap-file.\n\n    o The naming of the LTRANS linemap sections is stable so as not to\n      inhibit incremental LTO.\n\n    o Once the reader has processed the linemap sections, there is no\n      further overhead associated with inputting a location, unless it is an\n      adhoc location for the purpose of associating a discriminator or a\n      tree with the location. For the adhoc location case, the existing\n      lto_location_cache setup is still useful to avoid creating unneeded\n      adhoc locations, so this has been left in place as before. The\n      restriction that there be only one lto_location_cache at a time is no\n      longer strictly necessary; but it is still useful so that the IPA\n      passes can access the currently active one simply, so I have not\n      changed this for now.\n\nWith this new setup, PR65536 can be closed.  The subsequent patches in this\nseries will enable support for diagnostic pragmas.\n\ngcc/lto/ChangeLog:\n\n\tPR lto/65536\n\t* lang.opt: Add -fltrans-linemap-file.\n\t* lto-common.cc (lto_read_in_decl_state): Read the linemap ID from\n\tfunction sections.\n\t(linemap_section_id): New function.\n\t(create_subid_section_table): Note how many linemap sections were\n\tfound in each input section.\n\t(loc_map_decl_data): New GC root.\n\t(lto_file_read): Make ORDER into a static variable so it counts\n\tcontinuously across all files and sub-files provided on the command\n\tline.\n\t(lto_file_finalize): Handle reading the new LTRANS linemap file.\n\t(read_cgraph_and_symbols): Likewise. Also stop freeing\n\tALL_FILE_DECL_DATA at the end; the ordered list of files is now\n\tuseful later on for lto_copy_linemaps().\n\t* lto.cc (stream_out_linemaps): New function.\n\t(lto_wpa_write_files): Stream out the linemaps for LTRANS to use.\n\ngcc/testsuite/ChangeLog:\n\n\tPR lto/65536\n\t* gcc.misc-tests/outputs.exp: Adjust LTO -save-temps tests to expect\n\tthe new linemap file.\n\ngcc/ChangeLog:\n\n\tPR lto/65536\n\t* doc/lto.texi: Document the new LTO_section_linemap and the new\n\toption -fltrans-linemap-file.\n\t* lto-opts.cc (lto_write_options): Handle the new option.\n\t* opts.cc (gen_command_line_string): Likewise.\n\t* lto-section-in.cc (lto_section_name[]): Add new name for\n\tLTO_section_linemap.\n\t* lto-streamer-in.cc (get_location_from_idx): New function.\n\t(lto_location_cache::cmp_loc): Remove.\n\t(bp_unpack_delta): New function.\n\t(create_loc_map): New function.\n\t(get_loc_map): New function.\n\t(lto_location_cache::override_loc_map): New function.\n\t(lto_location_cache::apply_location_cache): Pervasive changes to\n\timplement new location streaming format.\n\t(lto_location_cache::input_location_and_block): Likewise.\n\t(lto_location_cache::input_location): Rename argument LOC -\u003e DEST\n\tfor clarity.\n\t(lto_read_body_or_constructor): Handle LINEMAP_ID in the decl state.\n\t(lto_data_in_create): Add NEED_LOCATION_CACHE argument. Rename local\n\tvariable DATA_IN to D to avoid clash with the type name.\n\t* lto-streamer-out.cc (clear_line_info): Update for change to class\n\toutput_block.\n\t(compute_map_hash): New function.\n\t(class location_output): New class.\n\t(location_output::record_location): New function.\n\t(bp_pack_delta): New function.\n\t(location_output::produce_linemap_section): New function.\n\t(lto_output_location_1): Pervasive changes to implement new location\n\tstreaming format.\n\t(copy_function_or_variable): Set the LINEMAP_ID in the decl state.\n\t(copy_linemap_section): New function.\n\t(copy_linemap_sections): New function.\n\t(lto_copy_linemaps): New function.\n\t(lto_register_linemap_for_output): New function.\n\t(lto_output_decl_state_refs): Output the LINEMAP_ID for function\n\tdecls.\n\t(lto_out_decl_state_written_size): Adapt for new LINEMAP_ID output.\n\t(produce_asm_for_decls): Output the linemap section when needed.\n\t* lto-streamer.cc (lto_get_section_name): Handle linemap sections,\n\twhich need an order suffix.\n\t* lto-streamer.h (enum lto_section_type): Fix typo in the\n\tcomment. Add LTO_section_linemap.\n\t(struct lto_loc_map): New struct.\n\t(class lto_location_cache): Pervasive changes to implement new\n\tlocation streaming format.\n\t(struct lto_in_decl_state): Add LINEMAP_ID member.\n\t(struct lto_out_decl_state): Likewise.\n\t(struct lto_file_decl_data): Add LOC_MAP_DECL_DATA, LOC_MAPS,\n\tand NUM_LINEMAP_SECTIONS members.\n\t(lto_linemap_output_id): New function.\n\t(struct output_block): Adjust members for new streaming format.\n\t(data_in::data_in): New function.\n\t(lto_data_in_create): Adjust prototype for new NEED_LOCATION_CACHE\n\targument.\n\t(lto_register_linemap_for_output): Declare.\n\t(lto_copy_linemaps): Declare.\n\t* lto-wrapper.cc (run_gcc): Pass new argument -fltrans-linemap-file.\n\t* timevar.def (TV_IPA_LTO_LINEMAP_IN): New timevar.\n\t(TV_IPA_LTO_LINEMAP_OUT): New timevar.\n\t(TV_IPA_LTO_LINEMAP_COPY): New timevar.\n",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "ed3180f38c01293d44852f2aa9d4afd5e9eb5f28",
      "old_mode": 33188,
      "old_path": "gcc/doc/lto.texi",
      "new_id": "3f3b326b4975b677162e1066ccbb2c66bace6214",
      "new_mode": 33188,
      "new_path": "gcc/doc/lto.texi"
    },
    {
      "type": "modify",
      "old_id": "a71c8a1bbd4881a6ce9993f9be8439c75c6f5f07",
      "old_mode": 33188,
      "old_path": "gcc/lto-opts.cc",
      "new_id": "ce3e79d845285b9f94731cb323cc08453440e9a5",
      "new_mode": 33188,
      "new_path": "gcc/lto-opts.cc"
    },
    {
      "type": "modify",
      "old_id": "c57f84f02bd46b66986dc94afe9e63107dadc3aa",
      "old_mode": 33188,
      "old_path": "gcc/lto-section-in.cc",
      "new_id": "04fafae027351220d0df8e23fa3a532551e2436f",
      "new_mode": 33188,
      "new_path": "gcc/lto-section-in.cc"
    },
    {
      "type": "modify",
      "old_id": "92f83ebbbb0dd25c7fc54236eb160a262166aa86",
      "old_mode": 33188,
      "old_path": "gcc/lto-streamer-in.cc",
      "new_id": "7906260af1994d898dc7f766ab46adf38801a61e",
      "new_mode": 33188,
      "new_path": "gcc/lto-streamer-in.cc"
    },
    {
      "type": "modify",
      "old_id": "87f9612b3d7937eccc48564e74395e92edadd11c",
      "old_mode": 33188,
      "old_path": "gcc/lto-streamer-out.cc",
      "new_id": "a88c1ee94f37fc42cd69e58b085e5ec68f49f0f0",
      "new_mode": 33188,
      "new_path": "gcc/lto-streamer-out.cc"
    },
    {
      "type": "modify",
      "old_id": "28b8cfc788edc8f467ad3c7bc381e3099b6d6322",
      "old_mode": 33188,
      "old_path": "gcc/lto-streamer.cc",
      "new_id": "078832fed4d734927905a8fb3ee3e0b66a6cefa2",
      "new_mode": 33188,
      "new_path": "gcc/lto-streamer.cc"
    },
    {
      "type": "modify",
      "old_id": "b09869195fc85a5739d52314e11ff66d1f665f75",
      "old_mode": 33188,
      "old_path": "gcc/lto-streamer.h",
      "new_id": "27126a4218526ed2479e1a1d61f2b54f7d1fdbc4",
      "new_mode": 33188,
      "new_path": "gcc/lto-streamer.h"
    },
    {
      "type": "modify",
      "old_id": "ca39f51e186350347f719329eb3dbfaaaf29bf4b",
      "old_mode": 33188,
      "old_path": "gcc/lto-wrapper.cc",
      "new_id": "bd25185c218e3dbfe83fca9e569d3f5a215994c1",
      "new_mode": 33188,
      "new_path": "gcc/lto-wrapper.cc"
    },
    {
      "type": "modify",
      "old_id": "16a2415f8c45f5d6a0e3a34c6caa3dab8d0ee02f",
      "old_mode": 33188,
      "old_path": "gcc/lto/lang.opt",
      "new_id": "bdced64c153a3ae8291222c189cc62b52223f5ab",
      "new_mode": 33188,
      "new_path": "gcc/lto/lang.opt"
    },
    {
      "type": "modify",
      "old_id": "16301516bd8a9ff82364a33c4f7d3339025c9fae",
      "old_mode": 33188,
      "old_path": "gcc/lto/lto-common.cc",
      "new_id": "96bd97f1374eb60a070baa15e6b8ade68732d0af",
      "new_mode": 33188,
      "new_path": "gcc/lto/lto-common.cc"
    },
    {
      "type": "modify",
      "old_id": "46f3799c4a04be173c92b04de804d4a8c74dae5a",
      "old_mode": 33188,
      "old_path": "gcc/lto/lto.cc",
      "new_id": "76837c27be81380ef174fbdf0ffbeef57b13217a",
      "new_mode": 33188,
      "new_path": "gcc/lto/lto.cc"
    },
    {
      "type": "modify",
      "old_id": "0d96fb6d78e577b01ce809de4aa39d484c02829b",
      "old_mode": 33188,
      "old_path": "gcc/opts.cc",
      "new_id": "88d533c664177fe60d1e6ae27d70c785016ca3c4",
      "new_mode": 33188,
      "new_path": "gcc/opts.cc"
    },
    {
      "type": "modify",
      "old_id": "0561dd58b60541ef10f0a3e9fe8dcaed3d56aef1",
      "old_mode": 33188,
      "old_path": "gcc/testsuite/gcc.misc-tests/outputs.exp",
      "new_id": "5a2d2f7fdbfacb69bc053cecee054507c6f29f57",
      "new_mode": 33188,
      "new_path": "gcc/testsuite/gcc.misc-tests/outputs.exp"
    },
    {
      "type": "modify",
      "old_id": "c70209976096be46d0d1d3a85819b3afef271988",
      "old_mode": 33188,
      "old_path": "gcc/timevar.def",
      "new_id": "fc78600b652d37212eb86f44fef774212d1fdf81",
      "new_mode": 33188,
      "new_path": "gcc/timevar.def"
    }
  ]
}