956:. The different classes of events that will be logged, as well as what details will appear in the event messages, are often considered early in the development cycle. Many event logging technologies allow or even require each class of event to be assigned a unique "code", which is used by the event logging software or a separate viewer (e.g., Event Viewer) to format and output a human-readable message. This facilitates localization and allows system administrators to more easily obtain information on problems that occur.
848:
965:
Difficulties in making a clear distinction between event logging and software tracing arise from the fact that some of the same technologies are used for both, and further because many of the criteria that distinguish between the two are continuous rather than discrete. The following table lists some
651:
Software tracing is a tool for developers to gather information for debugging. This information is used both during development cycles and post-release. Unlike event logging, software tracing usually does not have the concept of a "class" of event or an "event code". Other reasons why event-logging
833:
and often includes annotations and key-value attributes. Despite its widespread use, this design lacks rigorous justification and users frequently face challenges like missing features and confusing navigation. Alternatives to swimlane views exist, like Jaeger’s service dependency view or
656:
Because software tracing is low-level, there are often many more types of messages that would need to be defined, many of which would only be used at one place in the code. The event-code paradigm introduces significant development overhead for these "one-shot"
1066:
Given its low-level nature, tracing can generate a large volume of messages. To mitigate performance issues, it's often necessary to have the option to deactivate software tracing, either at the time of compilation or during run-time.
663:
Because the tracing output is intended to be consumed by the developer, the messages don't need to be localized. Keeping tracing messages separate from other resources that need to be localized (such as event messages) is therefore
834:
SkyWalking’s List, Tree, and Table views. Aggregate visualizations are also used for analyzing large volumes of traces, with systems like Canopy offering queryable metrics and Jaeger providing trace comparison features.
966:
important, but by no means precise or universal, distinctions that are used by developers to select technologies for each purpose, and that guide the separate development of new technologies in each area:
1056:, collecting trace data from a client's system hinges on their willingness and capability to install a version of the software specifically enabled for tracing, and subsequently replicate the issue.
826:
To model execution trees, ISVis converts a rooted tree into a directed acyclic graph while
Jinsight utilizes the call frame principle to gather and represent cumulative information about traces.
1049:
Enabling or disabling tracing during runtime often necessitates the inclusion of extra data in the binary. This can lead to performance degradation, even when tracing is not active.
553:
643:
Tracing is primarily used for anomaly detection, fault analysis, debugging or diagnostic purposes in distributed software systems, such as microservices or serverless functions.
507:
674:
Tracing messages should be kept in the code, because they can add to the readability of the code. This is not always possible or feasible with event-logging solutions.
569:
in software engineering refers to the process of capturing and recording information about the execution of a software program. This information is typically used by
474:
959:
Because event logging is used to log high-level information (often failure information), performance of the logging implementation is often less important.
807:
Trace generation of method calls can be done with source code instrumentation, runtime information collection, or under debugger control. Tracing macros,
546:
1211:
316:
1102:
812:
464:
1369:
539:
439:
180:
459:
1063:, not only in the accuracy and reliability of the trace output but also in ensuring that the process being traced remains uninterrupted.
869:
737:
497:
1356:. CASCON '04: Proceedings of the 2004 conference of the Centre for Advanced Studies on Collaborative research. IEEE. pp. 169–180.
406:
170:
1136:
1332:
939:
687:
273:
396:
391:
147:
920:
525:
892:
873:
416:
129:
109:
1060:
962:
A special concern, preventing duplicate events from being recorded "too often" is taken care of through event throttling.
326:
215:
205:
155:
1386:"A Qualitative Interview Study of Distributed Tracing Visualisation: A Characterisation of Challenges and Opportunities"
1000:
Must not be too "noisy" (containing many duplicate events or information that is not helpful for its intended audience)
899:
502:
230:
195:
46:
808:
321:
293:
858:
577:
purposes, and additionally, depending on the type and detail of information contained in a trace log, by experienced
444:
288:
175:
165:
104:
906:
877:
862:
743:
660:
The types of messages that are logged are often less stable through the development cycle than for event logging.
210:
190:
1112:
351:
220:
200:
888:
479:
361:
240:
114:
794:, tracing also requires special techniques for efficient instrumentation and logging and low CPU overhead.
611:
421:
331:
283:
225:
1021:
690:
open source project that provides comprehensive support for distributed tracing. Some vendors including
668:
590:
278:
245:
61:
51:
1239:
1076:
578:
341:
185:
119:
86:
66:
27:
623:
602:
336:
255:
81:
1416:
1353:
On the
Practical Feasibility of Software Monitoring: a Framework for Low-Impact Execution Tracing
1312:
1251:
1107:
992:
454:
1365:
1328:
1269:
1192:
1174:
913:
830:
791:
582:
1287:
1153:
Li, Bowen; Peng, Xin; Xiang, Qilin; Wang, Hanzhang; Xie, Tao; Sun, Jun; Liu, Xuanzhe (2022).
818:
Libraries used in source code send data to an agent or directly to the collection component.
1357:
1320:
1261:
1182:
1166:
631:
619:
615:
411:
374:
356:
346:
71:
1132:
1009:
829:
The primary visualization method is the swimlane view, which is exemplified by tools like
298:
250:
134:
56:
952:
Event logging provides system administrators with information useful for diagnostics and
1187:
1154:
709:
777:
is another tool in various operating systems for logging and tracing system messages.
1410:
1351:
1313:"Trace Recording for Embedded Systems: Lessons Learned from Five Industrial Projects"
1155:"Enjoy your observability: an industrial survey of microservice tracing and analysis"
762:
758:) where some of the technologies used to provide event logging may not be available.
683:
41:
1092:
722:
636:
614:
of a program (therefore excluding logging of data from an external source, such as
124:
1031:
Addition of new types of events, as well as new event messages, need not be agile
1324:
1385:
1080:
847:
1170:
652:
solutions based on event codes are inappropriate for software tracing include:
1265:
627:
570:
1361:
1273:
1178:
1097:
695:
586:
574:
76:
1212:"Council Post: Distributed Tracing: The Key To Microservices Observability"
1196:
1053:
953:
449:
401:
386:
381:
634:
events primarily of interest to a system administrator (see for example
784:
780:
755:
691:
988:
Logs "high level" information (e.g. failed installation of a program)
712:
have developed their own tracing frameworks namely Dapper and Canopy.
1079:, tracing data may include sensitive information about the product's
774:
770:
705:
699:
160:
1233:
1231:
1256:
761:
Linux offers system-level and user-level tracing capabilities with
766:
732:
235:
754:
In operating systems, tracing can be used in situations (such as
731:
Linux application tracing with UST – part of the same project as
469:
1345:
1343:
841:
1148:
1146:
1238:
Janes, Andrea; Li, Xiaozhou; Lenarduzzi, Valentina (2023).
787:
employ DTrace for tracing for the kernel and the userland.
1012:
output format is often desirable, sometimes even required
1059:
Tracing in software typically demands high standards of
1240:"Open tracing tools: Overview and critical comparison"
1311:Kraft, Johan; Wall, Anders; Kienle, Holger (2010),
1319:, Springer Berlin Heidelberg, pp. 315–329,
596:There is not always a clear distinction between
585:personnel and by software monitoring tools to
626:). Logs that record program usage (such as a
610:is almost never applied to logging that is a
547:
8:
991:Logs "low level" information (e.g. a thrown
980:Consumed primarily by system administrators
589:common problems with software. Tracing is a
876:. Unsourced material may be challenged and
728:Linux C/C++ application tracing with cwrap
554:
540:
18:
1255:
1186:
940:Learn how and when to remove this message
968:
640:) fall into a terminological gray area.
1124:
26:
1103:Instrumentation (computer programming)
508:Electrical and electronics engineering
1350:Mertz, Jhonny; Nunes, Ingrid (2019).
1052:If tracing is enabled or disabled at
7:
874:adding citations to reliable sources
773:also supports Linux kernel tracing.
738:Windows software trace preprocessor
1288:"Tracepoints (Debugging with GDB)"
702:also offer tracing SaaS services.
14:
1034:Addition of new tracing messages
1026:Localization is rarely a concern
1015:Few limitations on output format
983:Consumed primarily by developers
434:Standards and bodies of knowledge
1139:from the original on 2009-02-24.
846:
1244:Journal of Systems and Software
803:Trace generation and collection
526:Outline of software development
1159:Empirical Software Engineering
1:
1020:Event log messages are often
1325:10.1007/978-3-642-16612-9_24
815:techniques can be employed.
716:Application-Specific Tracing
16:Software debugging technique
1210:Mandel, Maya (2023-06-07).
809:Aspect-oriented programming
1433:
1171:10.1007/s10664-021-10063-9
1045:Challenges and limitations
744:Instruction set simulation
289:Software quality assurance
1266:10.1016/j.jss.2023.111793
1362:10.1109/SEAMS.2019.00030
1113:Observability (software)
667:There are messages that
274:Configuration management
1250:. Elsevier BV: 111793.
889:"Tracing" software
750:System-Specific Tracing
606:, except that the term
498:Artificial intelligence
612:functional requirement
422:Infrastructure as code
268:Supporting disciplines
591:cross-cutting concern
579:system administrators
279:Deployment management
1317:Runtime Verification
1077:proprietary software
1071:Security and privacy
870:improve this section
669:should never be seen
99:Paradigms and models
28:Software development
624:write-ahead logging
620:high-energy physics
600:and other forms of
22:Part of a series on
1133:"The Tracing Book"
417:Release automation
294:Project management
1371:978-1-7281-3368-3
1042:
1041:
975:Software tracing
950:
949:
942:
924:
792:embedded software
583:technical-support
564:
563:
455:ISO/IEC standards
1424:
1401:
1400:
1398:
1397:
1382:
1376:
1375:
1347:
1338:
1337:
1308:
1302:
1301:
1299:
1298:
1284:
1278:
1277:
1259:
1235:
1226:
1225:
1223:
1222:
1207:
1201:
1200:
1190:
1150:
1141:
1140:
1129:
969:
945:
938:
934:
931:
925:
923:
882:
850:
842:
725:'s trace command
647:Software tracing
632:operating-system
622:experiment, and
616:data acquisition
556:
549:
542:
503:Computer science
412:Build automation
19:
1432:
1431:
1427:
1426:
1425:
1423:
1422:
1421:
1407:
1406:
1405:
1404:
1395:
1393:
1384:
1383:
1379:
1372:
1349:
1348:
1341:
1335:
1310:
1309:
1305:
1296:
1294:
1286:
1285:
1281:
1237:
1236:
1229:
1220:
1218:
1209:
1208:
1204:
1152:
1151:
1144:
1131:
1130:
1126:
1121:
1089:
1073:
1047:
1010:standards-based
946:
935:
929:
926:
883:
881:
867:
851:
840:
824:
813:instrumentation
805:
800:
752:
718:
681:
649:
560:
531:
530:
521:
513:
512:
493:
485:
484:
435:
427:
426:
377:
367:
366:
312:
304:
303:
299:User experience
269:
261:
260:
151:
140:
139:
100:
92:
91:
37:
36:Core activities
17:
12:
11:
5:
1430:
1428:
1420:
1419:
1409:
1408:
1403:
1402:
1377:
1370:
1339:
1333:
1303:
1292:sourceware.org
1279:
1227:
1202:
1142:
1123:
1122:
1120:
1117:
1116:
1115:
1110:
1105:
1100:
1095:
1088:
1085:
1072:
1069:
1046:
1043:
1040:
1039:
1032:
1028:
1027:
1024:
1017:
1016:
1013:
1005:
1004:
1001:
997:
996:
989:
985:
984:
981:
977:
976:
973:
948:
947:
854:
852:
845:
839:
836:
823:
822:Trace Analysis
820:
804:
801:
799:
796:
763:kernel markers
751:
748:
747:
746:
741:
735:
729:
726:
717:
714:
680:
677:
676:
675:
672:
665:
661:
658:
648:
645:
562:
561:
559:
558:
551:
544:
536:
533:
532:
529:
528:
522:
519:
518:
515:
514:
511:
510:
505:
500:
494:
491:
490:
487:
486:
483:
482:
477:
472:
467:
462:
457:
452:
447:
445:IEEE standards
442:
436:
433:
432:
429:
428:
425:
424:
419:
414:
409:
404:
399:
394:
389:
384:
378:
373:
372:
369:
368:
365:
364:
359:
354:
349:
344:
339:
334:
329:
324:
319:
313:
310:
309:
306:
305:
302:
301:
296:
291:
286:
281:
276:
270:
267:
266:
263:
262:
259:
258:
253:
248:
243:
238:
233:
228:
223:
218:
213:
208:
203:
198:
193:
188:
183:
178:
173:
168:
163:
158:
152:
150:and frameworks
146:
145:
142:
141:
138:
137:
132:
127:
122:
117:
112:
107:
101:
98:
97:
94:
93:
90:
89:
84:
79:
74:
69:
64:
59:
54:
49:
44:
38:
35:
34:
31:
30:
24:
23:
15:
13:
10:
9:
6:
4:
3:
2:
1429:
1418:
1415:
1414:
1412:
1391:
1387:
1381:
1378:
1373:
1367:
1363:
1359:
1355:
1354:
1346:
1344:
1340:
1336:
1334:9783642166112
1330:
1326:
1322:
1318:
1314:
1307:
1304:
1293:
1289:
1283:
1280:
1275:
1271:
1267:
1263:
1258:
1253:
1249:
1245:
1241:
1234:
1232:
1228:
1217:
1213:
1206:
1203:
1198:
1194:
1189:
1184:
1180:
1176:
1172:
1168:
1164:
1160:
1156:
1149:
1147:
1143:
1138:
1134:
1128:
1125:
1118:
1114:
1111:
1109:
1106:
1104:
1101:
1099:
1096:
1094:
1091:
1090:
1086:
1084:
1082:
1078:
1070:
1068:
1064:
1062:
1057:
1055:
1050:
1044:
1037:
1033:
1030:
1029:
1025:
1023:
1019:
1018:
1014:
1011:
1007:
1006:
1003:Can be noisy
1002:
999:
998:
994:
990:
987:
986:
982:
979:
978:
974:
972:Event logging
971:
970:
967:
963:
960:
957:
955:
944:
941:
933:
922:
919:
915:
912:
908:
905:
901:
898:
894:
891: –
890:
886:
885:Find sources:
879:
875:
871:
865:
864:
860:
855:This section
853:
849:
844:
843:
838:Event logging
837:
835:
832:
827:
821:
819:
816:
814:
810:
802:
797:
795:
793:
788:
786:
782:
778:
776:
772:
768:
764:
759:
757:
749:
745:
742:
739:
736:
734:
730:
727:
724:
721:Tracing with
720:
719:
715:
713:
711:
707:
703:
701:
697:
693:
689:
685:
684:OpenTelemetry
678:
673:
670:
666:
662:
659:
655:
654:
653:
646:
644:
641:
639:
638:
633:
629:
625:
621:
617:
613:
609:
605:
604:
599:
594:
592:
588:
584:
580:
576:
572:
568:
557:
552:
550:
545:
543:
538:
537:
535:
534:
527:
524:
523:
517:
516:
509:
506:
504:
501:
499:
496:
495:
489:
488:
481:
478:
476:
473:
471:
468:
466:
463:
461:
458:
456:
453:
451:
448:
446:
443:
441:
438:
437:
431:
430:
423:
420:
418:
415:
413:
410:
408:
405:
403:
400:
398:
395:
393:
390:
388:
385:
383:
380:
379:
376:
371:
370:
363:
360:
358:
355:
353:
350:
348:
345:
343:
340:
338:
335:
333:
330:
328:
325:
323:
320:
318:
315:
314:
308:
307:
300:
297:
295:
292:
290:
287:
285:
284:Documentation
282:
280:
277:
275:
272:
271:
265:
264:
257:
254:
252:
249:
247:
244:
242:
239:
237:
234:
232:
229:
227:
224:
222:
219:
217:
214:
212:
209:
207:
204:
202:
199:
197:
194:
192:
189:
187:
184:
182:
179:
177:
174:
172:
169:
167:
164:
162:
159:
157:
154:
153:
149:
148:Methodologies
144:
143:
136:
133:
131:
128:
126:
123:
121:
118:
116:
113:
111:
108:
106:
103:
102:
96:
95:
88:
85:
83:
80:
78:
75:
73:
70:
68:
65:
63:
60:
58:
55:
53:
50:
48:
45:
43:
42:Data modeling
40:
39:
33:
32:
29:
25:
21:
20:
1394:. Retrieved
1392:. 2023-02-01
1389:
1380:
1352:
1316:
1306:
1295:. Retrieved
1291:
1282:
1247:
1243:
1219:. Retrieved
1215:
1205:
1162:
1158:
1127:
1093:Branch trace
1074:
1065:
1058:
1054:compile-time
1051:
1048:
1035:
964:
961:
958:
951:
936:
927:
917:
910:
903:
896:
884:
868:Please help
856:
828:
825:
817:
811:and related
806:
789:
779:
760:
753:
723:GNU Debugger
704:
682:
650:
642:
637:Event Viewer
635:
607:
601:
597:
595:
566:
565:
402:UML Modeling
397:GUI designer
62:Construction
52:Requirements
1390:IEEE Xplore
1081:source code
930:August 2024
571:programmers
120:Prototyping
115:Incremental
87:Maintenance
67:Engineering
1396:2024-01-12
1297:2022-06-24
1257:2207.06875
1221:2024-01-12
1119:References
1061:robustness
900:newspapers
798:Techniques
664:important.
628:server log
492:Glossaries
82:Deployment
1417:Debugging
1274:0164-1212
1179:1382-3256
1165:(1): 25.
1098:Debugging
1038:be agile
1022:localized
993:exception
857:does not
740:(aka WPP)
696:New Relic
657:messages.
575:debugging
311:Practices
135:Waterfall
110:Cleanroom
77:Debugging
47:Processes
1411:Category
1197:34867075
1137:Archived
1087:See also
954:auditing
587:diagnose
520:Outlines
450:ISO 9001
392:Profiler
387:Debugger
382:Compiler
357:Stand-up
1188:8629732
1108:Logging
914:scholar
878:removed
863:sources
785:SmartOS
781:FreeBSD
756:booting
692:Datadog
608:tracing
603:logging
598:tracing
567:Tracing
191:Lean SD
130:V model
72:Testing
1368:
1331:
1272:
1216:Forbes
1195:
1185:
1177:
916:
909:
902:
895:
887:
831:Jaeger
775:syslog
771:ftrace
706:Google
700:Splunk
465:SWEBOK
186:Kanban
161:DevOps
125:Spiral
57:Design
1252:arXiv
921:JSTOR
907:books
767:LTTng
733:LTTng
686:is a
679:Tools
630:) or
618:in a
460:PMBOK
375:Tools
236:SEMAT
231:Scrum
105:Agile
1366:ISBN
1329:ISBN
1270:ISSN
1193:PMID
1175:ISSN
1036:must
893:news
861:any
859:cite
783:and
765:and
710:Meta
708:and
688:CNCF
573:for
475:IREB
470:ITIL
440:CMMI
317:ATDD
226:SAFe
196:LeSS
171:DSDM
1358:doi
1321:doi
1262:doi
1248:204
1183:PMC
1167:doi
1075:In
872:by
790:In
581:or
480:OMG
407:IDE
362:TDD
352:SBE
342:DDD
327:CCO
322:BDD
246:TSP
241:TDD
221:RUP
216:RAD
211:PSP
206:MSF
201:MDD
181:IID
176:FDD
166:DAD
156:ASD
1413::
1388:.
1364:.
1342:^
1327:,
1315:,
1290:.
1268:.
1260:.
1246:.
1242:.
1230:^
1214:.
1191:.
1181:.
1173:.
1163:27
1161:.
1157:.
1145:^
1135:.
1083:.
1008:A
995:)
769:.
698:,
694:,
593:.
347:PP
337:CD
332:CI
256:XP
251:UP
1399:.
1374:.
1360::
1323::
1300:.
1276:.
1264::
1254::
1224:.
1199:.
1169::
943:)
937:(
932:)
928:(
918:·
911:·
904:·
897:·
880:.
866:.
671:.
555:e
548:t
541:v
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.