733:
invariance may be important, but it is not important in all settings. For instance, if the regressor variables can only take on integer values, then the choice of scale is not arbitrary, and so one might nor care whether one used a scale-invariant method or not. Also, if one knows a priori that the errors in dependent and independent variables have equal variance, then TLS gives a maximum likelihood solution. Rescaling the dependent and independent variables by different factors will make the error variances unequal, so it is not surprising that TLS no longer gives the maximum likelihood solution, and this doesn't imply that TLS is 'bad' or 'wrong'. It's just an answer to a particular question (or questions), which may or may not be the question
729:
incommensurability it is sometimes suggested that we convert to dimensionless variables - this may be called normalization or standardization. However there are various ways of doing this, and these lead to fitted models which are not equivalent to each other." This is stated as if it's a fatal flaw (at least that's how I read it), but it's really not. In particular, if one knows a priori that the errors in dependent and independent variables have equal variance and are all independent, then Draper and Smith show that total least squares gives the maximum likelihood solution for the regression coefficients.
74:
53:
279:" sign I presume means that the relationship is near perfect. As soon as that is not the case however, orthogonal regression goes wrong. Apparently this problem is not encountered much in computer science (??) but in other disciplines (econometrics, epidemiology, psychology, medicine, agricultural sciences, etc. etc.) it is more or less always the case that relationships to be estimated are imperfect.
22:
720:
there are errors in both independent and dependent variables are taken into account, and total least squares is only one of them. (The geometric mean functional relationship would be another, for instance.) Wouldn't it be more accurate to say that "Total least squares is a least squares data modeling technique in which the sum of the squared distances, as measured
370:
I think that's right. (Total least squares is a technique for error-in-variables that makes sense for the physical sciences (where you can plausibly know that two different variables have the same size of measurement error, but it makes less sense in the social sciences, where instrumental variables
227:
noise distribution is still not clear in any literature. Only thing we can say is that the residuals have even probability in positive and negative values. Based on that, noises in A and B have even distribution functions. If the real noise has odd pdf such as chi-distribution, then TLS may not work
724:
from the fit line to the data points, is minimized. Total least squares is often used in settings where there are observational errors in both the independent and dependent variables." This would emphasize that total least squares is a particular way of dealing with settings where there are errors
474:
Peter, please define your symbols. 'delta y', 'delta beta', 'K' and 'F', for instance. And it would be nice to see how the 'F' equations are condition statements (constraints). And what is meant by condition? Simultaneity? I find this article difficult to read and I can't imagine what a novice could
319:
In my view, "Orthogonal" and TLS are similar, but "Errors-in-variables" still leaves freedom in which direction to measure error. It all depends to what degree you would like to have the dependant variable contribute to error. So, in the case of the basic 2-dimensional model you are refering to, one
242:
This page is wrong. Error-in-variables refers to a model where the independent variables are measured with error. There are lots of possible solutions, of which total least squares is only one possibility (and one that's far from universally accepted. Economists would never use it, for example.)
732:
Then in the section "Scale invariant methods", it goes on to say that "In short, total least squares does not have the property of units-invariance (it is not scale invariant). For a meaningful model we require this property to hold." This last statement is just not true. In some settings, scale
728:
In the section "Geometrical interpretation", I think the first paragraph is excellent, but the second states that "A serious difficulty arises if the variables are not measured in the same units." and then goes on to describe the nature of this difficulty. It ends with "To avoid this problem of
719:
In particular, the first line states that "Total least squares is a least squares data modeling technique in which observational errors on both dependent and independent variables are taken into account." It seems that there are several different techniques that might be used in settings where
813:
Edit: I must be blind, it is actually mentioned at the top of the article "The total least squares approximation of the data is generically equivalent to the best, in the
Frobenius norm, low-rank approximation of the data matrix." However, I would appreciate this being elaborated as I presented
253:
I agree (that it's incomplete). In the eiv literature method of moments estimators (plural!) are more in use. Eg for the example on this page, see
Amemiya et al 1987. There are also several other estimators: SIMEX, estimating equations with adjusted score functions,...more? Libraries have been
343:
quantities. It is obvious from the discussion above that in other fields other methods are more likely to be used, but I have no experience of them. The earlier version mentions Data Least
Squares, and Structured Total Least Squares, but gave no details, so they have been omitted, for now.
228:
well. It is because TLS allows the negative correction in elements of A and b. Rather than speaking distribution, we can say that TLS minimizes the estimate of noise powers equally in A and b. I am sorry but I can describe only a very rough picture.
784:) makes it seem like TLS is simply augmenting the parameter space with one additional parameter attached to the output, y: minimize. Naturally there are an infinite number of solutions including all parameters set to zero. Which is why (
490:
This is inherently a very technical subject, not suitable for novices. "Some algebraic manipulation" is long and complicated, so I opted for a brief summary. As to whether TLS is innocent or not, that's for the jury to decide.
646:
I would like to know the differences between the total least square methods for the linear regression and the first eigenvector of the principal component analysis (or EOF). Anybody can explain it?
124:
918:
was being used for the design matrix in the first part, but then the matrix of unknown regression coefficients in the second. I hope with the change of lettering the transition is now easier.)
220:
Are the noises in A iid and the noises in b iid (but with possibly a differenct common distribution from the noises in A)-- OR, do the noises in A have the same distribution as the noises in b?
716:
I'm very happy this page exists, and I deeply appreciate the efforts of the various contributors, but I have to say that I find several aspects of this page confusing and/or misleading.
780:
I'm not an expert in this area, but I have a strong background with eigenvectors. I think it would be helpful, if possible, to rephrase the problem as an eigenvalue problem. Here (
190:
303:
I think we need some clarification on the terminology used throughout the literature. I am not a real statistician, but to me
Orthogonal regression as explained <A HREF="
906:
871:
277:
167:
210:
622:
588:
696:
There may also be the fact that SVD scales better to large numbers of degrees of freedom in which constructing the covariance matrix would be impractical...
526:
I'll add comments about what each line means, but I would love an explanation of why this works. I think I would understand SVD much better, for one thing.
389:
Error size has no meaning in physical measurements, unless it's of the same thing and in the same units. Only relative sizes matter. Is this what you mean?
946:
114:
951:
309:
is the same as TLS, at least it minimizes the same distances... could anyone elaborate? I think it would really add to the usability of the method.
821:
798:
762:
90:
284:
Besides that it is difficult to understand, this explanation seems very tailored to the treatment of this problem within one specific field.
941:
512:
662:
81:
58:
785:
781:
725:
in both the independent and dependent variables, not the only way, and would specify what is specific to total least squares.
259:
Carroll et al. heaviliy criticise orthogonal regression. In this specific example their criticism does not apply because the "
347:
Regarding the comment "this page is wrong", I would say rather that it's the title that is wrong. It should always have been
33:
788:) treats this as a problem of selecting the unit eigenvector with the smallest eigenvalue of the gram-matrix A'*A.
766:
747:
475:
make of it. (P.S. I agree with Walt Pohl on the absence of consensus on TLS. I am personally suspicious of it.)
825:
802:
516:
701:
687:
629:
531:
743:
658:
624:. I'm still confused why the singular values get ignored and where the negative sign comes from, though...
791:
I would greatly appreciate the article being explained this simply, if this description actually works.
339:. As it stands, it is mostly relevant to the physical sciences, where there is experimental error on all
289:
39:
841:
I've changed the variable letters used in the algebraic approach section so the equation solved is now
654:
310:
73:
52:
817:
794:
650:
674:
I'm not quite sure either; I'd love a good explanation. I think it has something to do with letting
21:
496:
443:
435:
431:
360:
914:(In particular, when reading the article as it previously was, it took me a while to realise that
172:
89:
on
Knowledge. If you would like to participate, please visit the project page, where you can join
697:
683:
625:
527:
480:
457:
420:
394:
376:
882:
847:
262:
149:
923:
786:
http://www.cs.princeton.edu/courses/archive/fall11/cos323/notes/cos323_f11_lecture09_svd.pdf
782:
http://www.cs.princeton.edu/courses/archive/fall11/cos323/notes/cos323_f11_lecture09_svd.pdf
761:
It would be nice if someone expanded the derivation so one could follow it more easily.
492:
439:
356:
320:
should not necessarily assume orthogonal distance meausurement as optimal solution. OK?
511:
Some comments for each line would be useful for those with no exposure to GNU Octave.
195:
935:
593:
559:
476:
453:
416:
390:
372:
336:
244:
415:, and start a new page on error-in-variables. Does anyone have any objections? --
919:
321:
927:
829:
806:
770:
751:
705:
691:
666:
633:
535:
520:
500:
484:
461:
447:
424:
411:
I was thinking (inspired by Peter Gans) that we should just move this page to
398:
380:
364:
324:
313:
292:
247:
232:
86:
911:
to more closely reflect the lettering used in the first part of the article.
548:
matrix—the analyzing matrix—basically captures the covariance structure of
229:
452:
Since no one else seems to have an opinion, I went ahead and did it. --
335:
This page has been revised to bring it in line with other articles on
304:
438:
to this article. That link would be better directed to the new page.
740:
Does anyone object to me editing the page to resolve these issues?
223:
To my best knowledge, the connection between the residuals and the
15:
430:
I would support that suggestion. I've just finished revising
590:
into the directions corresponding to the singular values of
544:
I looked at it more and expanded it. It looks like the
254:
written on this, and not all on "total least squares"!
885:
850:
596:
562:
265:
198:
175:
152:
299:
Total Least
Squares, Orthogonal, Errors-in-variables
85:, a collaborative effort to improve the coverage of
900:
865:
616:
582:
556:in that it transforms vectors right multiplied on
407:Should we move to his page to Total Least Squares?
271:
204:
184:
161:
8:
47:
884:
849:
603:
595:
569:
561:
264:
197:
174:
151:
49:
19:
642:Question about the total least squares
7:
351:as that is what it was about. Maybe
79:This article is within the scope of
305:http://www.nlreg.com/orthogonal.htm
38:It is of interest to the following
947:Mid-importance Statistics articles
176:
153:
14:
776:TLS with an eigenvector approach?
192:are ? And what is a 'residual in
99:Knowledge:WikiProject Statistics
72:
51:
20:
952:WikiProject Statistics articles
119:This article has been rated as
102:Template:WikiProject Statistics
611:
604:
597:
577:
570:
563:
1:
837:Algebraic approach (notation)
830:22:34, 12 February 2013 (UTC)
807:22:31, 12 February 2013 (UTC)
501:08:02, 22 February 2008 (UTC)
485:05:30, 22 February 2008 (UTC)
462:07:20, 21 February 2008 (UTC)
448:14:04, 20 February 2008 (UTC)
425:20:34, 16 February 2008 (UTC)
399:05:34, 22 February 2008 (UTC)
381:20:21, 16 February 2008 (UTC)
365:15:34, 16 February 2008 (UTC)
355:is another topic altogether?
248:02:44, 27 November 2006 (UTC)
185:{\displaystyle \Delta \beta }
93:and see a list of open tasks.
771:19:47, 9 February 2010 (UTC)
325:18:03, 5 December 2006 (UTC)
314:16:53, 5 December 2006 (UTC)
146:Can you please explain what
942:B-Class Statistics articles
901:{\displaystyle AX\approx B}
866:{\displaystyle XB\approx Y}
968:
233:01:40, 18 April 2006 (UTC)
928:16:12, 13 June 2013 (UTC)
752:17:50, 13 June 2009 (UTC)
521:08:35, 27 June 2008 (UTC)
434:where there is a link in
293:11:53, 5 April 2007 (UTC)
118:
67:
46:
706:14:18, 26 May 2009 (UTC)
692:13:58, 26 May 2009 (UTC)
667:07:01, 14 May 2009 (UTC)
634:14:18, 26 May 2009 (UTC)
536:02:57, 23 May 2009 (UTC)
272:{\displaystyle \approx }
162:{\displaystyle \Delta y}
712:Possible clarifications
682:be different shapes...
371:would be preferred. --
902:
867:
618:
584:
436:Underlying assumptions
273:
206:
186:
163:
142:This page is confusing
82:WikiProject Statistics
28:This article is rated
903:
868:
619:
585:
274:
207:
187:
164:
883:
848:
594:
560:
263:
196:
173:
150:
737:want an answer to.
470:Definitions, please
432:regression analysis
413:total least squares
353:Errors-in-variables
349:Total least squares
105:Statistics articles
898:
863:
614:
580:
269:
238:This page is wrong
202:
182:
159:
34:content assessment
820:comment added by
797:comment added by
670:
653:comment added by
205:{\displaystyle x}
139:
138:
135:
134:
131:
130:
959:
907:
905:
904:
899:
872:
870:
869:
864:
832:
809:
744:Adam Lyle Taylor
669:
647:
623:
621:
620:
617:{\displaystyle }
615:
607:
589:
587:
586:
583:{\displaystyle }
581:
573:
308:here </A: -->
278:
276:
275:
270:
211:
209:
208:
203:
191:
189:
188:
183:
168:
166:
165:
160:
125:importance scale
107:
106:
103:
100:
97:
76:
69:
68:
63:
55:
48:
31:
25:
24:
16:
967:
966:
962:
961:
960:
958:
957:
956:
932:
931:
881:
880:
846:
845:
839:
822:150.135.222.130
815:
799:150.135.222.130
792:
778:
763:203.167.251.186
759:
714:
698:—Ben FrantzDale
684:—Ben FrantzDale
648:
644:
626:—Ben FrantzDale
592:
591:
558:
557:
528:—Ben FrantzDale
509:
507:GNU Octave code
472:
409:
333:
301:
261:
260:
240:
218:
216:Can you clarify
194:
193:
171:
170:
148:
147:
144:
104:
101:
98:
95:
94:
61:
32:on Knowledge's
29:
12:
11:
5:
965:
963:
955:
954:
949:
944:
934:
933:
909:
908:
897:
894:
891:
888:
874:
873:
862:
859:
856:
853:
838:
835:
777:
774:
758:
755:
713:
710:
709:
708:
694:
643:
640:
639:
638:
637:
636:
613:
610:
606:
602:
599:
579:
576:
572:
568:
565:
539:
538:
513:217.169.50.138
508:
505:
504:
503:
471:
468:
467:
466:
465:
464:
408:
405:
404:
403:
402:
401:
384:
383:
332:
329:
328:
327:
300:
297:
296:
295:
290:87.219.191.214
286:
285:
281:
280:
268:
256:
255:
239:
236:
217:
214:
201:
181:
178:
158:
155:
143:
140:
137:
136:
133:
132:
129:
128:
121:Mid-importance
117:
111:
110:
108:
91:the discussion
77:
65:
64:
62:Mid‑importance
56:
44:
43:
37:
26:
13:
10:
9:
6:
4:
3:
2:
964:
953:
950:
948:
945:
943:
940:
939:
937:
930:
929:
925:
921:
917:
912:
895:
892:
889:
886:
879:
878:
877:
860:
857:
854:
851:
844:
843:
842:
836:
834:
833:
831:
827:
823:
819:
810:
808:
804:
800:
796:
789:
787:
783:
775:
773:
772:
768:
764:
756:
754:
753:
749:
745:
741:
738:
736:
730:
726:
723:
717:
711:
707:
703:
699:
695:
693:
689:
685:
681:
677:
673:
672:
671:
668:
664:
660:
656:
652:
641:
635:
631:
627:
608:
600:
574:
566:
555:
551:
547:
543:
542:
541:
540:
537:
533:
529:
525:
524:
523:
522:
518:
514:
506:
502:
498:
494:
489:
488:
487:
486:
482:
478:
469:
463:
459:
455:
451:
450:
449:
445:
441:
437:
433:
429:
428:
427:
426:
422:
418:
414:
406:
400:
396:
392:
388:
387:
386:
385:
382:
378:
374:
369:
368:
367:
366:
362:
358:
354:
350:
345:
342:
338:
337:least squares
330:
326:
323:
318:
317:
316:
315:
312:
306:
298:
294:
291:
288:
287:
283:
282:
266:
258:
257:
252:
251:
250:
249:
246:
237:
235:
234:
231:
226:
221:
215:
213:
199:
179:
156:
141:
126:
122:
116:
113:
112:
109:
92:
88:
84:
83:
78:
75:
71:
70:
66:
60:
57:
54:
50:
45:
41:
35:
27:
23:
18:
17:
915:
913:
910:
876:rather than
875:
840:
816:— Preceding
812:
811:
793:— Preceding
790:
779:
760:
742:
739:
734:
731:
727:
722:orthogonally
721:
718:
715:
679:
675:
645:
553:
549:
545:
510:
473:
412:
410:
352:
348:
346:
340:
334:
331:Page revised
302:
241:
224:
222:
219:
145:
120:
80:
40:WikiProjects
655:Tribute0708
649:—Preceding
311:Jeroenemans
936:Categories
757:Derivation
96:Statistics
87:statistics
59:Statistics
493:Petergans
454:Walt Pohl
440:Petergans
417:Walt Pohl
373:Walt Pohl
357:Petergans
245:Walt Pohl
818:unsigned
814:above.
795:unsigned
663:contribs
651:unsigned
477:Pgpotvin
391:Pgpotvin
341:measured
123:on the
30:B-class
920:Jheald
322:Witger
307:": -->
36:scale.
230:S. Jo
924:talk
826:talk
803:talk
767:talk
748:talk
702:talk
688:talk
678:and
659:talk
630:talk
552:and
532:talk
517:talk
497:talk
481:talk
458:talk
444:talk
421:talk
395:talk
377:talk
361:talk
225:real
212:' ?
169:and
735:you
243:--
115:Mid
938::
926:)
893:≈
858:≈
828:)
805:)
769:)
750:)
704:)
690:)
665:)
661:•
632:)
534:)
519:)
499:)
483:)
460:)
446:)
423:)
397:)
379:)
363:)
267:≈
180:β
177:Δ
154:Δ
922:(
916:X
896:B
890:X
887:A
861:Y
855:B
852:X
824:(
801:(
765:(
746:(
700:(
686:(
680:B
676:A
657:(
628:(
612:]
609:B
605:|
601:A
598:[
578:]
575:B
571:|
567:A
564:[
554:B
550:A
546:V
530:(
515:(
495:(
479:(
456:(
442:(
419:(
393:(
375:(
359:(
200:x
157:y
127:.
42::
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
