Criticism thread for my ratings methodology

destinationtruth
Posts: 806
Joined: Tue May 12, 2015 7:47 pm
Location: Cherokee Nation

RE: Criticism thread for my ratings methodology

Post by destinationtruth » Mon Oct 03, 2016 9:57 pm

<quote user="destinationtruth">
You do realize that just because it's an official site does not mean it is safe; for example look at hxxps://msu.edu/
This university site is currently blocked and found to be malicious by Emsisoft. https://www.virustotal.com/en/url/00a2fbe3f2a9e139e5a83d6b31e76dc166abe162f02489096828bf42435acbc2/analysis/1475525621/

Now most likely a FP (and I have submitted for evaluation to Emsisoft), yet still needs to be investigated before writing a blank check for "Trustworthy."

You have lot to learn grasshopper before you can walk across the rice paper.
[/quote]

The response from Emsisoft:

Heidi-Ann,

In looking at this again, there is still a malicious file hosted at the main msu.edu domain. If you can get me in contact with someone who would be responsible for the following address path, we can get this cleared up:

hxxp://www.msu.edu/software/xp/ ( I broke the link. HAF)

Unfortunately, while the file exists, we cannot remove it from blacklisting.

Best regards,

David Biggar
Chief Technical Support Officer

drsumit
Posts: 1584
Joined: Sun Jan 05, 2014 5:15 pm

RE: Criticism thread for my ratings methodology

Post by drsumit » Tue Oct 04, 2016 2:11 am

This is one perfect example by Heidi, - so we need to be more vigilant and cautious before writing off an official site as trustworthy

User avatar
Myxt
Posts: 4154
Joined: Sat Mar 05, 2011 6:18 am

RE: Criticism thread for my ratings methodology

Post by Myxt » Tue Oct 04, 2016 4:49 am

More about msu.edu - _https://www.virustotal.com/en/domain/msu.edu/information/

Example: Worm.Win32.Mabezat.b &nbsp; - do not access
msu.edu/software/xp/ KasperSky6.0%20Key.doc.exe

Fresh report: _https://www.virustotal.com/en/url/1ee274132321c211e3f2abeba8b20621211416b5495cb14355a5a387e55383db/analysis/1475555734/
About: _https://www.microsoft.com/security/portal/threat/encyclopedia/Entry.aspx?Name=Worm:Win32/Mabezat.B
____

Platinum ¬= power | status
Redundancy ¬= reliability
Broadbrush = dangerous
Isolation => ignorance

drsumit
Posts: 1584
Joined: Sun Jan 05, 2014 5:15 pm

RE: Criticism thread for my ratings methodology

Post by drsumit » Tue Oct 04, 2016 6:13 pm

An official site got hacked and malwares were planted onto the server

@Myxt - chances are that this file may have been removed by the site admins
because when I try to access that link (using virtual machines) its shows a 404
error.

User avatar
Myxt
Posts: 4154
Joined: Sat Mar 05, 2011 6:18 am

RE: Criticism thread for my ratings methodology

Post by Myxt » Wed Oct 05, 2016 12:00 am

I actually downloaded the file (sans added space char in the address) and tested it with local security to get its designation.

Many binaries have names like x.doc.exe, betting that users have set "Hide extensions for known file types", and won't notice .exe.

BCinUSA
Posts: 44
Joined: Thu Jul 28, 2016 12:42 pm

RE: Criticism thread for my ratings methodology

Post by BCinUSA » Wed Oct 05, 2016 1:09 am

Thread drift but I hope it's an appropriate place to ask. To me, there are 2 broad ways to rate a site-- based on the content the average (i.e. non-technical) web browsing person could view-- visible text, general theme and tone of the site, for example. And there's technical, metadata, hidden code, etc that many of the active forum posters here appear to be. Guess which category I'm in?

Is it WOT's view that raters should be expected to understand the more technical aspects of website development, or is it OK to rate based on the user's experience with the site and it's more easily visible content?

User avatar
Myxt
Posts: 4154
Joined: Sat Mar 05, 2011 6:18 am

RE: Criticism thread for my ratings methodology

Post by Myxt » Wed Oct 05, 2016 3:07 am

<quote user="bcinusa">
Thread drift but I hope it's an appropriate place to ask. To me, there are 2 broad ways to rate a site-- based on the content the average (i.e. non-technical) web browsing person could view-- visible text, general theme and tone of the site, for example. And there's technical, metadata, hidden code, etc that many of the active forum posters here appear to be. Guess which category I'm in?

Is it WOT's view that raters should be expected to understand the more technical aspects of website development, or is it OK to rate based on the user's experience with the site and it's more easily visible content?
[/quote]

Most people rate based on their end-user experience of content. WoT's guidelines are for everyone, regardless of technical abilities. Bring what you have to the table, follow the guidelines, subtract personal agendas, and your best will be good enough.

User avatar
Myxt
Posts: 4154
Joined: Sat Mar 05, 2011 6:18 am

RE: Criticism thread for my ratings methodology

Post by Myxt » Wed Oct 05, 2016 3:23 am

<speculation> Internet security may not be a top priority for schools which don't specialize in the subject. Unfortunately, altruism and lofty ideals are no defense against malicious actors, as demonstrated by recent ransomware attacks on hospitals. </speculation>

nova7
Posts: 507
Joined: Fri Apr 06, 2012 11:32 pm

RE: Criticism thread for my ratings methodology

Post by nova7 » Wed Oct 05, 2016 12:37 pm

<quote user="myxt">
... Unfortunately, altruism and lofty ideals are no defense against malicious actors, as demonstrated by recent ransomware attacks on hospitals.
[/quote]

Recent occurances of SEO spam at universities:
xHttp://news.softpedia.com/news/top-us-universities-hacked-and-injected-with-seo-spam-508700.shtml

Supported
Posts: 10
Joined: Fri Aug 19, 2016 9:56 am

RE: Criticism thread for my ratings methodology

Post by Supported » Thu Oct 06, 2016 4:17 am

Hope you all are doing well.

This post is separated into several sections, which shall tentatively consist of these four points.

1: Regarding MSU.edu specifically (digression)
2: Revisions to methodology guidelines made since the last post
3: Re-iterating certain details in Comment Keywords and Scoring Components
4: Relevant concerns pertaining to the possibility of including cyber-security components in site evaluations

Unfortunately, I do maintain a rating for msu.edu, the website of Michigan State University, so I feel obligated to spend a couple of words on this matter. But before I get into it, I would like to reiterate the following.

Respect statement: No matter what the nature of the replies are, I will do my best to treat them with the utmost respect and consideration. In my book, you all are the most prestigious members of this sector of the World Wide Web.

I apologize if this reply seems lengthy, dirty, disorganized, or just plain grammatically incorrect in any way; I am covering a larger spectrum of replies this time around, and it is currently late at night.

Disclaimer: this post may be retroactively edited for quality assurance purposes.

==================================================================

Regarding MSU.edu Specifically (feel free to ignore if this seems irrelevant to the topic):

<quote user="destinationtruth">
You do realize that just because it's an official site does not mean it is safe; for example look at hxxps://msu.edu/
This university site is currently blocked and found to be malicious by Emsisoft. [hxxps]://www.virustotal.com/en/url/00a2fbe3f2a9e139e5a83d6b31e76dc166abe162f02489096828bf42435acbc2/analysis/1475525621/

Now most likely a FP (and I have submitted for evaluation to Emsisoft), yet still needs to be investigated before writing a blank check for "Trustworthy." as you did.

You have lot to learn grasshopper before you can walk across the rice paper.
[/quote]

First and foremost, thank you very much for bringing this to my attention.

Maybe we can start by looking at the URL itself, which is:

hxxps://msu.edu/software/xp/ + KasperSky6.0%20Key.doc.exe

I did some research on the file in question and on the particular circumstances of this breach, and what I was able to find seemed to reflect what is indicated here:

hxxps://moz.com/researchtools/ose/links?site=https%3A%2F%2Fmsu.edu%2Fsoftware%2Fxp%2FKasperSky6.0%2520Key.doc.exe&filter=&source=all&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=

For comparison, see:

For the URL, hxxps://msu.edu/academics/colleges.html,
>>> the report at hxxps://moz.com/researchtools/ose/links?site=https%3A%2F%2Fmsu.edu%2Facademics%2Fcolleges.html&filter=&source=all&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=

For the URL, hxxp://www.lib.msu.edu/,
>>> the report at hxxps://moz.com/researchtools/ose/links?site=lib.msu.edu&filter=&source=all&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=

For the root domain URL,
>>> the report at hxxps://moz.com/researchtools/ose/links?site=https%3A%2F%2Fmsu.edu%2F&filter=&source=all&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=

That is, there currently appear to be no hyperlinks from publicly-accessible web pages that point to this file. The URL itself appears to be reasonably isolated.

With this information, from what I know about the case so far, I may only be able to conclude that the file is available solely through the navigation of directory browsing, by way of only the following particular addresses: hxxps://msu.edu/software/, hxxps://msu.edu/software/xp/, hxxps://msu.edu/software/w2000/, hxxps://msu.edu/software/nt/, and hxxps://msu.edu/software/mswin/.

To be specific, under the definition of ‘Website’ currently in use (listed above), contents accessible exclusively via directory browsing, or that are single-landing pages, web applications, 3xx redirected addresses, or downloadable files, cannot be covered in my evaluations. On the other hand, if there was a link to this file or any other malicious file on a web page of this site, be it a standard HyperText Markup link, Javascript onClick Event, Adobe Flash Hotspot, Microsoft Silverlight HyperLinkButton object, etc., as long as said link is embedded on a publicly-available, non-directory browsing, web page, I would fully consider it as within the scope of my evaluation.

Also; MSU.edu has a Symantec Trust Seal, an SSL certificate from Internet2, and is aggregately rated 93 for Trustworthiness, 91 for Child Safety, with WOT Confidences of 5/5 and 4/5, respectively. I am not sure there is a point to scoring against the grain, or against Symantec Corporation, or against Internet2 (University Corporation for Advanced Internet Development).

In any case, thank you all very much for pointing this out and introducing questions into the matter. Because of this, for clarity purposes, I have amended several of the definitions under my Methodology sections.

==================================================================

Revisions to methodology guidelines made since the last post

a: As mentioned before, because of questions introduced regarding the technical security of certain higher education websites, I have revised several of the definitions listed under my Methodology to include the following clause:

“Any part or component of a rated site that does not fall under this definition is not covered in the evaluation.”

<quote user="notbuyingit">
When they submit ratings without doing additional research, WOT users who rely almost entirely upon DMOZ.org, similar directories, blacklists or whitelists are expected to identify their sources…
[/quote]

b: While I do not ever submit ratings without doing additional research, I now appended a library of sources I have consulted during the information-gathering and cross-checking processes of my evaluations. As noted in the relevant section, they are not the exclusive determinants of my final opinions. Review my methodology statement to find this addition.

c: I have revised the ‘Reason’ definition under ‘My Ratings’ to include the following clause:

“Others may be applied, to be included as additional reasons only, if deemed necessary.”

==================================================================

Re-iterating certain details in Comment Keywords and Scoring Components

<quote user="drsumit">
This is one perfect example by Heidi, - so we need to be more vigilant and cautious before writing off an official site as trustworthy
[/quote]

a: The ‘Trustworthiness’ component of the site’s score under My Ratings is influenced by the ’Trustworthy’ keyword, but is not equal to it. ‘Trustworthiness’ is formally influenced by a combination of factors, most heavily the presence of the ‘Official’ keyword and the ‘Trustworthy’ keyword in the scorecard comment, but also an assorted sum of other elements, which may or may not include more relatively subjective criteria such as presentation/ease-of-navigation, clarity of information, aesthetics, etc.

b: Regarding the ‘Trustworthy’ and ‘Reasonably Trustworthy’ keywords, I think it may be convenient to understand certain specifics.

There currently exist three possibilities for this slot (Trustworthy, Reasonably Trustworthy, or N/A). The official definition of this parameter as it currently stands is:

“The institution faithfully provides the services it claims to provide, in both nature and quantity.”

For this singular parameter, its definition does not cover anything about the site itself, but rather focuses entirely on the institution behind it.

With that said, it may be arguable that since the intent of an educational institution is to… educate, that becoming involved in the distribution of malware may somehow impede upon that purpose. In view of this, I may, based on this criterion and depending on the severity of the circumstance, adjust my comment for that parameter down one or two steps. However even in doing so, it will most likely amount to a relatively minor portion of the larger-picture evaluation of the ‘Trustworthy’ keyword, which itself already hinges upon a number of factors, as explained more assiduously in the methodology, so while this may change the Scorecard Comment, it may not ultimately have a large effect on the ‘Trustworthiness’ score under My Ratings.

Also, this is necessarily in keeping with the assumption that a malicious file was not, for some reason, published for educational purposes to begin with, in service of a Computer Science course not unlike Introduction to Malware (CSEC-466-01), a three-credit offering at the Rochester Institute of Technology’s B. Thomas Golisano College of Computing and Information Sciences, as described here hxxps://www.rit.edu/academicaffairs/tigerterms/introduction-malware, or Introduction to Malware Analysis (DEF 4601P), a course worth 1.40 CEUs at Georgia Tech Professional Education, as described here hxxps://pe.gatech.edu/courses/introduction-malware-analysis, for instance.

c: Since the ‘Reason’ component of My Ratings contains within it a provision for ‘Malware or viruses’, I have appended the definition of that attribute to include the possibility of addressing these problems, accordingly.

==================================================================

Relevant concerns pertaining to the possibility of including cyber-security components in site evaluations

I may have an Early Proposal for a revised Scorecard Comment format, to be applied both prospectively and retroactively across all of my ratings, drafted over the next several weeks. This, in addition to the ‘Reason’ component of My Ratings, will be what are largely responsible for communicating site cyber-security issues, should they happen to exist, in the future.

However, in considering this, I may need to find ways of properly addressing the following issues, which have plagued both site owners and myWOT users in the past, and which I imagine continue to do so today.

_________________________________________________________________

a: Up-to-dateness
There are more complaints about this problem than any other. Obviously there are the threads that I am sure everyone is aware of, such as hxxps://www.mywot.com/en/forum/32952-mass-rating-tool-madness/page-2?comment-190736=, and hxxps://www.mywot.com/en/forum/59942-gurbl-should-be-banned.

However beyond that, what I’m referring to is the practicality of manually ensuring up-to-dateness across a body of several thousand ratings that include as a part of them findings for a subject that can prove incredibly transient. Often (as in this example), it is the case of adding/changing/removing a single file or process that makes or breaks a site’s security status, and when it happens most likely I would not even know about it. By the time I am eventually informed, my evaluation could already have been several years outdated (as in here hxxps://www.mywot.com/en/scorecard/msu.edu#comment-86508997).

Given this, I cannot be completely sure that within my scope, if in my evaluation placing any significant emphasis on the cyber-security of a site is indeed reasonably achievable.

Is there or can there be the possibility that a faster, perhaps more informed, system of self-correction for ratings might exist?

_________________________________________________________________

b: Definition of “Not Safe”

<quote user="destinationtruth">
You do realize that just because it's an official site does not mean it is safe;
[/quote]

This particular concern may foray into the realm of subjectivity, but requires addressing nevertheless. Assuming that the content-in-violation shall fall under my scope, and that cyber-safety may eventually be factored into the various applicable parts of my evaluation that I have mentioned, should a site’s reputation be affected by its immediate current status, by its record of previous attacks and the frequency of them, or both? To allegorize this, should an individual be judged based on whether he/she is currently at large for an offense, on the contents of his/her police record, or both? And if the answer is the second or third option, how many strikes would it take to permanently damage the overall evaluation?

To cough up an example (for the sake of securing some mutual agreement I am using the federal domain hxxp://fbi.gov), consider this report:
hxxps://exchange.xforce.ibmcloud.com/url/fbi.gov

one will note the following details.

<quote user="ibm x-force exchange on its website – exchange.xforce.ibmcloud.com">

Worm.Mytob.NK, 558639F353EC65DD47F4BCC9264BDE38, Domain specified as sender for spam, Attachment: document.zip, Sep 22, 2015 9:30 AM

Worm.Mytob.NK, D0880DE7B9951CE37CFBEC84A1C342C8, Domain specified as sender for spam, Attachment: doc.zip Jul 20, 2015 9:30 AM

Worm.Mytob.NK, 77F4FA8C38FB524EB28B986BD595FFA5, Domain specified as sender for spam, Attachment: doc.zip Jun 27, 2015 11:30 AM

Worm.Mytob-73, 2E1DFB845939BCC1CCC9DFA25C7245E8, Domain specified as sender for spam, Attachment: message.zip, Apr 5, 2015 10:30 PM

X-Force Malware Family Report Worm.Mytob-73, Win.Worm.Mytob-399
TABLE_SUMMARY_MALWARE_PAGE
First seen Feb 17, 2015 8:45:00 PM
Last seen Mar 23, 2016 4:30:00 AM
[/quote]

This is compared to what currently would be used as a metric by most experienced members:
hxxps://www.virustotal.com/en/url/c3b9924751022fa13dd4e40a80debf8796291b6663cfb0ab8a63fef043637fca/analysis/1475716929/

which, if you know where I’m going with this, obviously displays

<quote user="virustotal on its website – virustotal.com">
URL: [hxxp]://fbi.gov/
Detection ratio: 0 / 68
Analysis date: 2016-10-06 01:22:09 UTC ( 0 minutes ago )
[/quote]

What would a WOT member be conscientiously responsible for reporting in regards to this domain? Should this example, despite its current clean bill of health, retain a demoted reputation for having a record of security breaches and multiple associations with malware over a proven period of time?

As of now, the site has a 92 Trustworthiness rating with a WOT Confidence of 5/5, and a 93 Child safety rating with a WOT Confidence of 5/5.

I suppose I have to keep reminding myself of:

<quote user="mywot team">
Remember your responsibility
Every member is responsible for their own ratings and comments. Ratings can especially affect small businesses and in some cases, people’s lives, so try to be reasonable and do not give a rating without a good reason.
[/quote]

So while keeping in mind:

<quote user="destinationtruth">
You have lot to learn grasshopper before you can walk across the rice paper.
[/quote]

Hopefully I can learn soon, because if I am to venture into this area, I do not want to be opening a can of worms, and in order to avoid doing just that, I feel like I should be much more informed than how I currently am right now. I guess I’m supposed to ask the question: what should generally be considered grounds for formally accusing a site of posing a security flaw on myWOT? What should be considered “safe”, and what should be considered “not… safe”?

_________________________________________________________________

c: Impact on Personal Conscience
I’m afraid of opening any doors I can’t close. I’m afraid of the possibility of unfairly damaging or otherwise attacking the reputation or business of an institution that is likely both innately well-intentioned and the innocent victim of some unrelated, malicious party.

==================================================================

In closing:

<quote user="super hero!">
I strongly believe that a comment should be the responsibility of that member and her or his free opinion of that site as in my case…
[/quote]

<quote user="myxt">
Platinum ¬= power | status…
Isolation => ignorance
[/quote]

<quote user="nova7">
Recent occurances of SEO spam at universities:
xHttp://news.softpedia.com/news/top-us-universities-hacked-and-injected-with-seo-spam-508700.shtml
[/quote]

It is relieving to see these replies being expressed out of genuine concern, and it is why I hope to continually work to improve the substance of my evaluations looking forward.

To reiterate, I may have an Early Proposal for a revised Scorecard Comment format, to be applied both prospectively and retroactively across all of my ratings, drafted over the next several weeks. I look forward to the prospect of sharing it with you all.

In the meantime, I apologize for the long, if convoluted, post, and thank you very much for your replies, and any others that have been posted so far. As I am still relatively new to this forum, I would indeed greatly appreciate any suggestions and/or constructive criticism that you all may have to offer, and will be reading everything carefully and communicating as best I can.

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests