Experimental Study of the Validity of Sociodiagnostic Methods
V. L. Talanov
Experimental Study of the Validity of Sociodiagnostic Methods
First and foremost, the present validity study concerned the NSI, NSIC, NSIE, IOSG, and ISG forms of sociodiagnostic questionnaires, constructed according to V. Talanov’s methodology and used in 2013 for offline diagnosis of type and other socionic parameters in the Internet projects of A. Khizhnyak and S. Sibirskaya [1; 2]. In accordance with the agreement on the use of the author’s diagnostic methodology, the detailed primary data from this anonymous offline diagnosis were provided by A. Khizhnyak and S. Sibirskaya to the author – with the right to use, analyze, and publish them.
The number of specifically diagnostic questions in the named questionnaires varies, ranging from 340 to 580 (some “non-diagnostic” questions were included in the questionnaires for research purposes), but despite differences in the size of the diagnostic block, the calculated sensitivity of all the questionnaires is approximately the same (the questions were selected so that, in questionnaires with fewer diagnostic questions, their mean diagnostic sensitivity is higher, compensating for the smaller number of questions). The questions in all five questionnaires overlap very little (coinciding diagnostic questions in any arbitrarily selected pair of questionnaires constitute less than 10% of the diagnostic block, that is, all 5 of the named questionnaires can quite reasonably be regarded as instruments independent of one another).
In addition, the study incidentally yielded results concerning the validity of a number of other sociodiagnostic questionnaires (by other authors), as well as face-to-face typings performed by various socionics workshops. All these results will be presented in this article.
By Way of Preface: General Theory Important for What Follows. Calculation of the Probability of a Chance Coincidence of Two Erroneous but Independent Sociotype Diagnoses
This calculation will be very useful to us in other sections of the article.
First, let us introduce notation. The letters P1 and P2 will denote the probability of correct (true) diagnosis by two methods (№1 and №2). The letters D1 and D2 will denote the sample-average accuracy of each of these methods (the proportion of correct diagnoses obtained for them in a sample of limited size). If the sample is sufficiently large, then P1=D1 and P2=D2, - we will make this substitution below without stating it separately each time.
We will denote the proportion of simultaneous accurate (correct) diagnosis by both methods as DD(1;2). In contrast to DD(1;2), we will call the quantity СХ the empirical concordance, equal to the empirical proportion of matches in the sample between the diagnoses of two different methods (it exceeds DD(1;2) by the probability of a simultaneous identical error by the two methods).
Suppose that two different sociodiagnostic procedures simultaneously made an error in diagnosing a sociotype (and this is already the established fact from which we can proceed further). There is only one “correct” type for the subject, while there are as many as 15 incorrect types to choose from (all except the one correct type, which is excluded in advance from the possible realizations of the event). If errors in determining the sociotype were purely random, then the probability that independent errors by two methods would simultaneously coincide (in the finally selected type) would equal 1/15=0,067=6,7% (because, for the second method to arrive at the same erroneous type as the first, it has a completely equal and random choice among 15 alternatives).
But the probability that a sociodiagnostic method (whether a questionnaire, face-to-face typing, or self-typing – it makes no difference) will err toward one or another “incorrect” type is by no means the same for different types of the socion that stand in different relations to the “correct” type: for example, errors toward “quasi-identicals” or “kindred types” occur much more frequently than toward “activators” or “supervisors.” This narrows the field of randomness in the choice of one type error or another, so the resulting probability that the names of the erroneously diagnosed type will coincide for two sociodiagnostic methods, even if in every other respect their errors are completely independent of one another, will nevertheless clearly be higher than 1/15. But how much higher? This can be determined not approximately but with complete precision by mathematical-statistical modeling, based on the extensive statistics available to us on the frequencies of different diagnostic errors (that is, the frequencies of erroneous diagnosis as a kindred type, look-alike type, quasi-identical, dual, conflictor, etc.).
Omitting the calculations, let us give the result. It turns out that the “effective” number of independent random alternatives for the sociotype “selected” by an error is not 15¸ but on average about 9. The reduction in the “effective” number of alternatives from 15 to 9 is explained by the reduction in the number of degrees of freedom when selecting a possible error – and this occurs, among other things, because of the obvious probabilistic inequality of type errors that distinguish the diagnosed type from the true type either by a single basic dichotomy or by a larger number of them.
Therefore, the probability that independent errors by two methods will simultaneously coincide (in the finally selected type) is not 1/15=0,067, but 1/9=0,111 (because, for the second method to arrive at the same erroneous type as the first, it has a completely equal and random choice from an effective number of 9 alternatives that are, as it were, equal in weight, instead of 15 alternatives with differing probabilities of realization).
The concordance of two independent diagnostic procedures exceeds their joint accuracy by the proportion of erroneous diagnoses that also coincided by chance for the two methods. The probability that the first method makes an error is (1-D1). The probability that the second method makes an error is (1-D2). If the methods are independent, then the probability of their simultaneous error (regardless for the moment of whether the errors are identical or different) equals the product of the two probabilities, namely (1-D1)*(1-D2). Given the established fact that both sociodiagnostic methods made an error, the mean probability that their error is IDENTICAL IN THE SELECTED TYPE equals 1/9=0,111 (as we showed above). Thus, to obtain the final probability of two errors, one must also multiply by this probability, giving:
**Probability of a joint identical error by two methods = (1-D1)(1-D2)0,111
Probability of their simultaneous absolutely correct (true) diagnosis = D1*D2
Hence the empirical concordance of the two methods (the total proportion of actual, that is, random or non-random, matches between their socionic type diagnoses in the sample):
*СХ(1;2)=D1D2 + (1-D1)(1-D2)0,111 {1};
Knowing the diagnostic concordance of two methods in a large sample and one of the accuracies D (for example, D1), one can therefore determine the unknown accuracy D2:
D2=(СХ(1;2)-0,111+0,111D1)/(D1+0,111D1-0,111) {2}. For example, for СХ(1;2)=0,49 and D1=0,7, we obtain D2=0,4567/0,6667=0,685 (and by no means 0,7, as would be the case if the condition СХ(1;2)=D1*D2 were satisfied).
If both accuracies of two independent methods are unknown, but for some reason they are assumed to be equal in magnitude, then, replacing both unknown accuracies with the single letter D, we obtain: СХ(1;2)=DD+0,111(1-D)*(1-D)
Hence (also remembering that 0,111=1/9), we obtain:
D=0,1(1+3square root(10*СХ(1;2)-1))** {3}. For example, for СХ(1;2)=0,64 this gives D1=D2=D=0,797; for concordance СХ(1;2)=0,25 it gives D=0,467, etc.
1. Repeat Diagnosis of Socionic Parameters (with a Substantial Time Gap Between Two Testings) Using Another, Independent Questionnaire Form (with a Completely Different Set of Questions)
Of the total 2500 questionnaires, we had 127 cases in which the same respondents, in order to recheck their sociotype, voluntarily underwent type diagnosis using another questionnaire form after several days (or, in approximately half of the cases, several weeks), that is, with a substantial time gap and using different diagnostic questions. The linear correlation coefficients between the type profiles obtained from two different questionnaires (for 127 pairs of cases), as well as the concordance of diagnoses of the “leading sociotype” from questionnaires of two different forms in this sample (namely, the proportion of exact matches), are presented below in Table 1.1.:
Table 1.1. Correlation coefficients of type profiles and concordance of type diagnoses (proportion of exact matches) when typing with a time gap (from several days to several weeks) using two different questionnaires (with different questions)
| All pairs of repeat questionnaires (N=127 pairs) | Pairs of repeat questionnaires only with positive reliability scores (on average for the pair) (N=106 pairs) | |
|---|---|---|
| Concordance of two sociotype diagnoses (proportion of exact matches in diagnosis between the first and second questionnaire forms when socionic testing with them was separated in time by several days to several months) | 0,709 | 0,745 |
| Calculated – according to formula {3} – mean accuracy of a single typing by questionnaire | 0,840 | 0,862 |
| Linear correlation coefficient between two type profiles - the initial and repeat profiles (each profile consists of 16 numbers characterizing the similarity of the respondent’s character to each of the 16 sociotypes). Before calculating the correlation between them, each profile is standardized by contrast (reduced to a standard deviation of one within the array of the 16 numbers constituting it) | 0,838 | 0,866 |
| Mean shared variance of the two type profiles, initial and repeat (= the square of the correlation coefficient) | 0,70 | 0,75 |
| Proportion of variance in the complete socionic profile (including information not only about the leading type but also about all secondary accentuations toward it) that is not reproduced on repeat testing. It is associated both with objective variations in the respondent’s mental state and with errors of questionnaire typing. | 0,30 | 0,25 |
When analyzing Table 1.1., it should be taken into account that respondents who undergo repeat typing generally have borderline types. This means the superimposition on the main type of an additional accentuation toward some other type (or several of them at once), which reduces the difference in height between the leading peak of the type profile and the peaks following it in magnitude. A reduction in the height difference between two peaks, in turn, reduces the reproducibility of the diagnosis of the leading profile peak on repeat testing. Thus, the figures in Table 1.1. should be regarded as a lower bound for the mean estimate of the reproducibility of the diagnosis (and of profile shapes) in repeat testing using questionnaires with different questions.
Let us also note that the mean proportion of matches between two discrete diagnoses (concordance) and the shared variance of two type profiles turn out to be practically equal to one another (namely, =0,75 for questionnaires with positive profile reliability). Accordingly, the mean accuracy of a single measurement of type by questionnaire (=0,86) likewise practically coincides with the correlation coefficient between the arrays of two type profiles, initial and repeat. This result is not accidental - according to the laws of mathematical statistics, these coincidences are regular and are what should be obtained.
Of interest in repeat questionnaire diagnosis (using questionnaires with a different set of questions) is the reproducibility not only of the leading sociotype and, overall, of the respondent’s complete socionic type profile (also reflecting secondary accentuations), but also of other results of questionnaire typing. In particular, of interest is the reproducibility of the reliability indicator (rigidly associated with the contrast of the resulting type profile) and the reproducibility of the tendency of the respondent’s answers toward the intertype mean (the tendency to answer questionnaire questions in the way that an average representative of the socion, averaged across all types, would answer them). The latter indicator is partly associated with the respondent’s conformity or possible haste in filling out the questionnaire, but in part it also characterizes the objective smoothness of the respondent’s type and functional profiles. Another indicator whose reproducibility is of interest is an objective indicator of hysteroid-demonstrative character traits (it is unrelated to the content of the respondent’s answers to the questionnaire questions and is calculated exclusively from their formal aspect, namely, from the tendency to agree more often with any statements offered for “trying on” than to reject them). The association of this last indicator with hysteroid-demonstrative character traits is due to the fact that, as it turns out, in substantive personality analysis it correlates only with these character traits, and does so very strongly. The point is that it is precisely people with a demonstrative behavioral radical who have the highest suggestibility, hence their tendency, when trying statements on themselves, to agree with them more often.
For all three of the named control indicators, their correlation with analogous indicators on repeat testing using a questionnaire of another form is shown in Table 1.2.:
Table 1.2. Reproducibility of certain questionnaire control indicators during repeat typing (on average after 2 weeks) using a questionnaire with different questions
| Linear correlation coefficient between indicator values (in a sample of 127 people, comparing initial and repeat testing) | |
|---|---|
| Conditional reliability indicator (equal to a linear function of the contrast of the resulting type profile – before equalizing profile contrasts by normalizing them to a standard deviation of one) | 0,72 |
| Indicator of correlational coupling of the respondent’s answers with the averaged answers of the “average person” | 0,51 |
| Extra-questionnaire indicator of hysteroid-demonstrative character traits (calculated not from the substantive but from the formal features of questionnaire responses) | 0,81 |
2. Repeat Diagnosis of Socionic Parameters with a Substantial Time Gap, but Using a Sociodiagnostic Questionnaire of the Same Form (with the Very Same Questions).
In our sample of 2500 questionnaires, there were 80 pairs of questionnaires, or 80 respondents, meeting this requirement (that is, they filled out the same questionnaire twice, with a time interval of one week or more, without replacing the new completion by editing the previous questionnaire).
By analogy with the preceding Section 1, we present Tables 2.1. and 2.2., reflecting the reproducibility on repeat testing of the same indicators as in the preceding section of the article, but with repeat testing using not a different questionnaire form, but the very same form (that is, with the same questions):
Table 2.1. Correlation coefficients of type profiles and concordance of type diagnoses (proportion of exact matches) during repeat typing with a time gap (from several days to several weeks) using a questionnaire with the same questions
| All pairs of repeat questionnaires, including 20 pairs with negative profile reliability (total N=80 pairs) | |
|---|---|
| Concordance of two sociotype diagnoses (proportion of exact matches in diagnosis between the first and second questionnaire administrations) during repeat administration after a period from several days to three weeks) | 0,775 |
| Calculated – according to formula {3} – mean accuracy of a single typing by questionnaire (under the conditional assumption of independence of the two questionnaires, which in this case is INCORRECT). The diagnostic accuracy calculated by the formula is therefore overestimated. | 0,879 |
| Linear correlation coefficient between two type profiles - the initial and repeat profiles (each profile consists of 16 numbers characterizing the similarity of the respondent’s character to each of the 16 sociotypes). Before calculating the correlation between them, each profile is standardized by contrast (reduced to a standard deviation of one within the array of the 16 numbers constituting it) | 0,905 |
| Mean shared variance of the two type profiles, initial and repeat (equal to the square of the correlation coefficient) | 0,82 |
| Proportion of variance in the complete socionic profile (including information not only about the leading type but also about all secondary accentuations toward it) that is not reproduced on repeat testing. It is associated both with objective variations in the respondent’s mental state and with the influence on responses of a random-situational factor | 0,18 |
Table 2.2. Reproducibility of certain questionnaire control indicators during repeat typing using the same questionnaire about 10 days later
| Linear correlation coefficient between indicator values (in a sample of 80 people, comparing initial and repeat testing) | |
|---|---|
| Conditional reliability indicator (equal to a linear function of the contrast of the resulting type profile – before equalizing profile contrasts by normalizing them to a standard deviation of one) | 0,86 |
| Indicator of correlational coupling of the respondent’s answers with the averaged answers of the “average person” | 0,89 |
| Extra-questionnaire indicator of hysteroid-demonstrative character traits (calculated not from the substantive but from the formal features of questionnaire responses) | 0,87 |
3. Mean Concordance of Questionnaire Diagnosis with the Results of Face-to-Face Expert Diagnosis of Sociotypes
A. Khizhnyak previously published results on the concordance of sociotype diagnosis by the NSI questionnaire with the results of face-to-face expert type diagnosis previously performed for questionnaire respondents at the Ufa Socionics Club, as well as among regular participants of the SHS forum, at the Kaliningrad Center of Socionics, and at the Yekaterinburg Socionics Club (http://socionics.kiev.ua/f/index.php?showtopic=1549&st=50).
These results are summarized below in Table 3.1.:
Table 3.1.
| Ufa Socionics Club | Regular participants of the SHS forum | Kaliningrad Center of Socionics | Yekaterinburg Socionics Club | |
|---|---|---|---|---|
| Total number of respondents (questionnaires) examined | 13 | 9 | 25 | 19 |
| Mean concordance of diagnoses by questionnaire and face-to-face typing (proportion of exact matches between the two diagnoses of the leading sociotype) | 0,77 | 0,89 | 0,60 | 0,58 |
| Number of questionnaires with positive reliability scores (a sufficiently high, that is, sufficiently contrasting, type profile according to the questionnaire) | 12 | 8 | 21 | 17 |
| Mean concordance of diagnoses in the subsample of questionnaires with positive reliability | 0,83 | 0,88 | 0,67 | 0,65 |
If the results of Table 3.1. are summed across all columns, we obtain Table 3.2., from which certain conclusions can be drawn about the comparative accuracy of questionnaire diagnoses in two groups of questionnaires: those with positive and those with negative conditional reliability (questionnaires in the second group have low-contrast type profiles, that is, profiles that are highly noisy and weakly expressed in height):
Table 3.2.
| All 4 socionics schools combined that provided questionnaires from their subjects for comparison of sociotype diagnoses | |
|---|---|
| Total number of respondents (questionnaires) examined | 66 |
| Mean concordance of diagnoses by questionnaire and face-to-face typing (proportion of exact matches between the two diagnoses of the leading sociotype) | 0,667 |
| Calculated – according to formula {2} – mean accuracy of a single face-to-face typing in the four socionics workshops (taking into account a mean questionnaire typing accuracy =0,84) | 0,79 |
| Number of questionnaires with positive reliability scores (a sufficiently high, that is, sufficiently contrasting, type profile according to the questionnaire) | 58 |
| Mean concordance of diagnoses in the subsample of questionnaires with positive reliability | 0,726 |
If we use the results of Table 1.1., from which it follows that the mean accuracy of exact questionnaire determination of sociotype for all subjects is 84,2%, and for the subsample of subjects with positive questionnaire reliability – 86,3%, then we can also estimate the mean accuracy of diagnoses obtained by face-to-face typing (for the four indicated socionics schools in weighted-average form). For the overall array of all subjects it is about 79% (approximately equal to 0,667/0,842), and for the subsample of subjects with positive reliability of their questionnaires it is 84% (approximately equal to 0,726/0,863). In the second case the result is somewhat higher (84% correct diagnoses in face-to-face typing), apparently because subjects with positive reliability of sociodiagnostic questionnaires have either a higher intelligence quotient or longer socionics experience, so their face-to-face typing is easier to perform and gives a more accurate result.
We also see, from the example of the four socionics clubs presented (whose face-to-face type diagnosis, along with Vera Novikova’s socionics school, is possibly among the most reliable of Russia’s socionics schools), that their mean type-diagnostic accuracy (79%) is nevertheless lower than the mean type-diagnostic accuracy using diagnostic questionnaires according to Talanov’s methodology (84% accurate hits, see Table 1.1.).
At the next stage of the study, comparison of diagnoses “by questionnaire” and by the results of face-to-face typing in various socionics centers was continued, but in this case the assistance of the centers was no longer used. Instead, all respondents whose questionnaires arrived for processing over the Internet “on their own initiative” were asked, in addition to their own assessment of their sociotype, also to indicate its diagnosis as assigned by some socionics center during face-to-face type diagnosis (if such diagnosis had ever taken place). Of the total 2400 respondents whose questionnaires arrived for processing “on their own initiative,” 155 people reported their type according to the results of face-to-face typing (also indicating the center where this typing had been performed). Unfortunately, there is very great diversity among sociodiagnostic centers and individual practicing socionists who perform face-to-face typing. It is possible to identify only about five socionics centers or individual socionists whose number of clients following face-to-face typing exceeds 10 people in our sample. Vera Novikova’s “Socionics Practicum” (Moscow) proved the most popular – 25 of its clients (including both face-to-face typing performed for them and typing via Skype) entered our sample. There were also 14 people who had undergone face-to-face type diagnosis at Elena Udalova’s School of Applied Socionics (Moscow); 12 people who had received a sociotype diagnosis at sessions of the Open Socionics Club of St. Petersburg; 10 people who had undergone typing by Sergey Ionkin (also known as “Enman,” Moscow-Noginsk); 12 people who had undergone face-to-face typing at Vladimir Mironov’s “Dynamic Socionics” (8 of them in St. Petersburg and 4 people in other cities where branches of this firm have been opened, - but we combine their results because the typing methodology used by this firm is common, based on a distinctive interpretation of the substantive content of the Reinin traits). For all other socionists, socionics schools, and workshops, the number of respondents in our sample who had undergone face-to-face typing with them was less than 10 in each case, so we will not consider them.
The results of the analysis of concordance between diagnoses from face-to-face typing and from the questionnaire are shown in the following Table 3.3. It should, however, be taken into account that the composition of respondents differs between Tables 3.1. and 3.3. in that, in the latter case, the sample consisted of questionnaires submitted on their own initiative, and therefore the percentage of respondents who initially disagreed with the diagnosis assigned to them during face-to-face typing may be artificially elevated – precisely this circumstance may provide a reason to additionally complete a rather long and labor-intensive sociodiagnostic questionnaire. Therefore, in Table 3.3. (compared with Table 3.1., where the workshops themselves asked their subjects to complete questionnaires), the percentage of respondents who had definitely received an incorrect sociotype diagnosis during face-to-face typing at socionics workshops may be inflated (relative to the true statistical mean).
Table 3.3.
| Open Socionics Club of St. Petersburg | Vera Novikova’s Socionics Type Laboratory (or Socionics Practicum) | Sergey Ionkin (“Enman”) | School of Applied Socionics (Elena Udalova) | “Dynamic Socionics” (Vladimir Mironov) | All socionists combined (52 socionics workshops) that performed face-to-face typing of 154 respondents in the sample | |
|---|---|---|---|---|---|---|
| Total number of respondents who, in addition to completing the questionnaire, also underwent face-to-face typing at the corresponding socionics workshop | 12 | 25 | 10 | 14 | 12 | 154 |
| Mean concordance of diagnoses by questionnaire and face-to-face typing (proportion of exact matches between these two sociotype diagnoses) | 0,50 | 0,56 | 0,20 | 0,14 | 0,08 | 0,34 |
| Mean concordance of the diagnosis received by the respondent during face-to-face typing at a socionics workshop with the respondent’s self-diagnosis (with self-perception, self-typing) | 0,45 | 0,58 | 0,50 | 0,36 | 0,58 | 0,55 |
| Mean concordance of the questionnaire diagnosis with the respondent’s self-diagnosis (with self-perception, self-typing) | 0,81 | 0,63 | 0,60 | 0,43 | 0,25 | 0,52 |
| Number of respondents with positive questionnaire reliability scores | 19 | 10 | 117 | |||
| Mean concordance of questionnaire and face-to-face typing diagnoses in the subsample of questionnaires with positive reliability | 0,72 | 0,20 | 0,35 |
Before analyzing Table 3.3., it should be emphasized that sample sizes of 10-14 people (and even 25 people) are certainly insufficient for precise comparative conclusions about the accuracy of sociodiagnosis in one socionics workshop or another. One can speak only of statistical tendencies (which, in larger samples, may with sufficiently high probability be confirmed, but may also be refuted, although with lower probability).
Thus, a high level of agreement between diagnoses “from face-to-face typing” and “from the large sociodiagnostic questionnaire” characterizes face-to-face typings in Vera Novikova’s Socionics Type Laboratory and in the Open Socionics Club of St. Petersburg. The level of concordance with the questionnaire in these two cases does not statistically differ from that of the socionics workshops previously considered in Table 3.1., while the level of accuracy of diagnoses issued during face-to-face typing belongs to a confidence interval (p<0,1) of approximately 60% to 86% accuracy. We emphasize that this concerns sociotypes within the framework of the traditional socionics evidently followed by most socionists in Russia and Ukraine. Those schools that, for example, proceed in sociodiagnosis from all 15 Reinin traits and their own distinctive conceptions of their substantive content (including V. Mironov’s school) may not fit within these canons, which may be what the result of Table 3.3. indicates. We do not know what accounts for the low concordance results in Udalova’s column, but this result may also be a consequence of insufficient sample size. The comparatively moderate matches of several of Ionkin’s face-to-face diagnoses (he is the author of a number of very interesting studies in socionics) with the questionnaire diagnoses may likewise be an artifact of the small sample, but may, unfortunately, also result from his recent enthusiasm for certain Reinin traits whose substantive content has not been fully verified, as a result of which the more reliable holistic properties of type are underestimated.
As another merely statistical tendency, the increase in the degree of “authoritarian persuasiveness” of socionics workshops also attracts attention - from left to right in Table 3.3. (from the first data column to the fifth column, excluding the final sixth). The conclusion about this tendency follows from the fact that, along this sequence, the influence of the type declared by the school to the subject during face-to-face diagnosis (especially when declared incorrectly) on the subject’s own conception of their sociotype increases. Thus, the effect of socionic suggestion, a kind of socionic “zombification” (implemented through a skillfully imposed diagnosis), increases from left to right along this sequence.
The first two data columns in Table 3.3. confirm the validity of diagnosis both by the sociodiagnostic questionnaires of the series under discussion (according to Talanov’s methodology) and by face-to-face typing at the two socionics workshops corresponding to these columns. The expected value of the mean accuracy of diagnoses “from face-to-face typing” at these workshops is about 59% for the Open Socionics Club of St. Petersburg, and about 66% for Vera Novikova’s workshop (this is the result of calculation using formula {2}, taking into account the previously demonstrated mean questionnaire accuracy of 84%). The actual levels of mean accuracy of the “face-to-face” diagnoses at these workshops may, however, be higher (we recall that a feature of the sample used to obtain the results in Table 3.3. is the respondents’ desire to recheck their sociotype – that is, there may be an intentionally elevated accumulation in it either of persons with an objectively “borderline” type or of those who, for some reason, were assigned an incorrect diagnosis during face-to-face typing).
The last, sixth column of Table 3.3. is discouraging. It shows that, on average, taking into account all currently practicing socionists who take it upon themselves to perform face-to-face sociodiagnosis (they are located mainly in Russia), the mean concordance of their diagnoses with the more accurate diagnoses from the “large” questionnaire is only 34%, and, taking into account the mean accuracy of questionnaire diagnoses of 84%, the accuracy of all face-to-face socionic diagnosis in Russia turns out to average only 39% (calculation by formula {2}, taking into account the questionnaire accuracy value of 84% shown above). Thus, in more than fifty percent of cases, subjects are given an incorrect type as a result of face-to-face diagnosis. And this is mainly among socionists who charge money for their services. A mean diagnostic accuracy of 39% for face-to-face typing corresponds to concordance of diagnoses from different “average socionists” at approximately the level of 19-20% - approximately this value had also been confirmed earlier, before our work, by other authors in studies of cross-concordance of diagnoses in the typing of celebrities. And in the well-known Dnepropetrovsk experiment of 1999, СРТ-99, only a slightly higher mean concordance of type diagnoses by different professional typers was obtained when they performed face-to-face typing of 13 “guinea-pig” subjects - namely 27%, which corresponds to a mean typing accuracy of 49% [4].
At the same time, Tables 3.1. and 3.3. show that there are also socionics workshops (and apparently quite a few of them) in which the mean accuracy of face-to-face sociotype diagnosis has been raised almost to the maximum theoretical limit, namely to 80-84%. The concordance of diagnoses between such schools should be 60-70%.
When we speak of the upper theoretical limit of the accuracy of determining a sociotype, we mean that, according to our previous studies, the theoretical limit of mean sociodiagnostic accuracy is approximately 90% (it is the same for questionnaires and for face-to-face typing). The existence of this limit is due to the fact that about 15-20% of all people have an almost strictly intermediate type between two “standard” types of the socion (such is the distribution of their socionic functions by magnitude). Therefore, assigning this percentage of people strictly to one of the types can in principle be done only entirely arbitrarily, on a purely random basis, and in no other way – especially taking into account the inevitable fluctuations in a person’s profile of mental functions depending on state and mood. If the range of these situational fluctuations exceeds the difference in height between the two leading peaks of a person’s type profile (even if such a small difference exists), then no matter how sensitive the diagnostic method may be, an unambiguous accurate determination of a person’s type in a single testing becomes impossible - both in the case of questionnaires and in face-to-face typing).
4. Analysis of Cross-Concordance of Diagnoses from the “Large” IOSG Sociodiagnostic Questionnaire (According to Talanov’s Methodology) with Respondents’ Self-Typing Results and with Diagnoses from Sociodiagnostic Questionnaires by Other Authors.
Svetlana Sibirskaya previously published on this topic [3]. Below, in Tables 4.2. and 4.3., we present the results of cross-concordance among six sociotype diagnoses (see their numbering in Table 4.1.), obtained from the final and more complete data provided by S. Sibirskaya to the author for analysis. In addition to the increase in sample size, we also present the analysis results in a somewhat different processing than in S. Sibirskaya’s publication.
Note: respondents entered their type according to self-typing in the protocol AFTER they had taken tests 3, 4, 5, and 6 – that is, already taking the results of these tests into account (and undoubtedly under their influence). The results of Gulenko’s test and, in part, Tangemann’s and the “Piter Test” (V. Mironov et al.) had especially strong influence, as they were more authoritative (Ferzen’s test, despite its objectively higher reliability compared with Tangemann’s test and the “Piter Test,” influenced the self-typing results less, as detailed statistical calculations showed).
The large IOSG diagnostic questionnaire, which performs diagnosis not by individual bimodal traits but by the holistic properties of sociotypes, was in the overwhelming majority of cases completed by respondents last, and they did not know its results. Therefore, self-typing did not depend on its results. For this reason, the diagnosis from the IOSG questionnaire is fully independent with respect to the other methods. However, the self-typing result cannot be regarded as strictly independent with respect to tests 3, 4, 5, and 6. Partial dependence of self-typing on their diagnoses (due to correlated errors) may lead to overestimation of the corresponding pairwise concordances. Tests 3, 4, and 5 are also not, in the strict sense, independent of one another, because all of them appeal only to the 4 basic Jungian traits as the basis for diagnosis. Therefore, the errors made by these tests may also be directionally biased in advance (for example, with weakly expressed irrationality and strongly expressed decisive traits in some extraverted ethical-intuitive type, all these tests will tend to classify the respondent as IEE, whereas in fact the respondent is EIE, etc.). Therefore, the empirical values of pairwise concordances among these tests may (and most likely will) also be overestimated.
Table 4.1. Diagnostic methods used in S. Sibirskaya’s project (for cross-comparison of the TIM diagnosis). Test addresses are given according to the links on S. Sibirskaya’s website page containing the first part of the project: “comparison of socionics questionnaires.”
| 1 | IOSG-579 test | |
| 2 | Comprehensive self-typing (respondents’ self-assessment of sociotype) | |
| 3 | Gulenko test | http://www.tests-tests.com/gulenko.php?qnum=1&question=0 |
| 4 | Ferzen test | http://www.socionictest.net/ |
| 5 | Tangemann test | http://aeterna.qip.ru/test/view/103109/ |
| 6 | Piter Test | http://vikrob.narod.ru/vovka1.htm (test of the “Working Group on Socionics” - with participation by V. Mironov et al., 2002) |
Table 4.2.
Number of non-empty combinations of two diagnoses (number of pairwise comparisons)
| 1 | 2 | 3 | 4 | 5 | 6 | |
|---|---|---|---|---|---|---|
| 1 | 182 | 201 | 198 | 199 | 198 | |
| 2 | 182 | 182 | 181 | 180 | 181 | |
| 3 | 201 | 182 | 198 | 199 | 198 | |
| 4 | 198 | 181 | 198 | 196 | 198 | |
| 5 | 199 | 180 | 199 | 196 | 196 | |
| 6 | 198 | 181 | 198 | 198 | 196 |
Table 4.3.
Concordance (proportion of matching diagnoses)
| 1 | 2 | 3 | 4 | 5 | 6 | |
|---|---|---|---|---|---|---|
| 1 | 0,62 | 0,51 | 0,40 | 0,33 | 0,20 | |
| 2 | 0,62 | 0,52 | 0,48 | 0,36 | 0,21 | |
| 3 | 0,51 | 0,52 | 0,42 | 0,32 | 0,19 | |
| 4 | 0,40 | 0,48 | 0,42 | 0,29 | 0,18 | |
| 5 | 0,33 | 0,36 | 0,32 | 0,29 | 0,12 | |
| 6 | 0,20 | 0,21 | 0,19 | 0,18 | 0,12 |
Table 4.4.
| Method of obtaining the socionic diagnosis | Calculated type-diagnostic accuracy of the method (taking into account formula {2} and IOSG accuracy =84%, calculated on the basis of Table 1.1.) | Calculated accuracy of the methods assuming a lower value of the mean accuracy of the IOSG method (=75%) | |
|---|---|---|---|
| 1 | IOSG-579 test | 0,84 | 0,75 |
| 2 | Comprehensive self-typing (respondents’ self-assessment of their sociotype) | 0,734 | 0,821 |
| 3 | Gulenko test | 0,602 | 0,671 |
| 4 | Ferzen test | 0,470 | 0,521 |
| 5 | Tangemann test | 0,376 | 0,414 |
| 6 | Piter Test | 0,218 | 0,234 |
It should be noted that, in the full sample (2500 respondents), the mean concordance of self-typing with sociotype diagnosis by the NSI, NSIC, NSIE, IOSG, and ISG questionnaire forms was 58%. Based on a mean accuracy of questionnaire diagnoses of 84%, this concordance corresponds to a mean self-typing accuracy of 68,4% (0,684), so it is unlikely that in this subsample of 200 people it could have amounted to more than 73-75%.
Main Conclusions:
For the NSI, NSIC, NSIE, IOSG, and ISG questionnaire forms constructed according to V. Talanov’s methodology, their mean probability of correctly determining the subject’s leading type (in the full sample of subjects with any values of questionnaire-profile accuracy) is, according to the results of Table 1.1., about 84%. In the subsample of subjects having only positive (in conditional points) reliability scores for the type profiles diagnosed by the questionnaire, this probability of a correct diagnosis averages 86%.
The mean accuracy of self-typing results in a sample of 2500 people was about 68-69%, but in some subsamples (if these are people with better-than-average familiarity with socionics) this value can increase substantially.
The Gulenko test has a mean diagnostic accuracy no higher than 65%, the Ferzen test – no higher than 50%, the Tangemann test – no higher than 40%, and the “Piter Test” – no higher than 23%.
The mean diagnostic accuracy of face-to-face typing (averaged over all typing acts performed within Russia and, in part, Ukraine and Belarus by persons engaged in face-to-face typing on a professional basis) is, based on the data of Table 3.3., about 39-40% (conversion of the empirically found mean concordance with the questionnaire, equal to 34%, into mean typing accuracy using formula {2}). The calculated concordance of diagnoses among such “average socionists” should be about 16%. However, in some socionics workshops, the mean accuracy of their face-to-face typing rises to 60-70 and even to 80%. Among these socionics schools, interschool diagnostic concordance should rise to the level of 50-65%. Among the commercial socionics centers that we were able to evaluate, the Kaliningrad Center of Socionics and Vera Novikova’s Moscow “Socionics Practicum” showed the best results. And among non-commercial socionics workshops, we have already been able to establish a high or sufficiently high level of accuracy of face-to-face typings at the Ufa, Yekaterinburg, and St. Petersburg socionics clubs.
The mean accuracy of socionic type diagnosis (if the entire human population is considered as its sample) has an upper theoretical limit of approximately 90%, corresponding to an upper theoretical limit of diagnostic concordance of approximately 80%. The upper limit of diagnostic accuracy is due to the fact that about 15%-20% of all people have an almost intermediate type between two or even three “standard” types of the socion (and the intermediate character of their type, repeatedly confirmed when testing with a wide variety of methods, corresponds in them to a distinctive hierarchy of the 8 functions of the psyche). Confidently assigning this portion of people specifically to one of the 16 types of the socion can only be done arbitrarily, without regard to their actual psychological qualities. The issue of intermediate (borderline) cases is considered in greater detail and with more evidence in our other works, both past works and those still being prepared for publication. Here we will merely note that the reason for the fundamental impossibility of unambiguous type diagnosis of people with an approximately borderline type lies in the inevitable fluctuations in a person’s profile of mental functions depending on state and mood (if the range of these situational fluctuations exceeds the difference in height between the two leading peaks of the person’s type profile, then, no matter how sensitive the diagnostic method may be, unambiguous accurate determination of type in a single testing becomes impossible, both in the case of questionnaires and in face-to-face typing).
Literature and References:
- NSIC and NSIE diagnostic questionnaires (link as of December 2013): http://hizhnjak.narod.ru/index.html
- ISG diagnostic questionnaire (link as of December 2013): http://grafol.ru/swet/cot.html
- S. Sibirskaya. The Myth of Low Concordance Has Been Dispelled. - http://grafol.ru/swet/cmc2_rezult.html
- V. L. Pavlov, N. L. Malaya, I. E. Semencha, V. I. Gudramovich, S. E. Petsold (Dnepropetrovsk). What the СРТ-99 Experiment Showed. - http://vlp.chat.ru/srtrez1.html
- Concordance of Typings. - http://zanoza.socioland.ru/index.php/%D0%A1%D1%85%D0%BE%D0%B4%D0%B8%D0%BC%D0%BE%D1%81%D1%82%D1%8C_%D1%82%D0%B8%D0%BF%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%B8%D0%B9
St. Petersburg, 14.12.-28.12.2013
copyleft: V. L. Talanov, 2013 (the author permits free reproduction of this article on any scientifically oriented Internet pages – provided that the URL of the original source is indicated: http://www.sociotoday.narod.ru/validnost.htm)