Specialized medical search-engines are no better than general search-engines in sourcing consumer information about androgen deficiency

D. Ilic1,2, T.L. Bessell1, C.A. Silagy* and S. Green1

1 Monash Institute of Health Services Research, Monash University, Clayton 3168, Australia

2 To whom correspondence should be addressed at: Monash Institute of Health Services Research, Locked Bag 29, Monash Medical Centre, Clayton VIC 3168, Australia. e-mail: dragan.ilic{at}med.monash.edu.au


    Abstract
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
BACKGROUND: The Internet provides consumers with access to online health information; however, identifying relevant and valid information can be problematic. Our objectives were firstly to investigate the efficiency of search-engines, and then to assess the quality of online information pertaining to androgen deficiency in the ageing male (ADAM). METHODS: Keyword searches were performed on nine search-engines (four general and five medical) to identify website information regarding ADAM. Search-engine efficiency was compared by percentage of relevant websites obtained via each search-engine. The quality of information published on each website was assessed using the DISCERN rating tool. RESULTS: Of 4927 websites searched, 47 (1.44%) and 10 (0.60%) relevant websites were identified by general and medical search-engines respectively. The overall quality of online information on ADAM was poor. The quality of websites retrieved using medical search-engines did not differ significantly from those retrieved by general search-engines. CONCLUSION: Despite the poor quality of online information relating to ADAM, it is evident that medical search-engines are no better than general search-engines in sourcing consumer information relevant to ADAM.

Keywords: ADAM/androgen deficiency/Internet/quality of information


    Introduction
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
The Internet is a continually expanding communications network with more than 530 million users worldwide, accessing an estimated 300 million web pages (Computer Industry Almanac, 2002Go; accessed on July 13, 2002; Lawrence and Giles, 1998Go). It provides consumers with continuous access to information presented in a multimedia format. Information once exclusive to health professionals is now readily attainable by anyone with Internet access (Jadad and Gagliardi, 1998Go).

Internet access to medical information, support networks and the purchase of medications online provides consumers with the opportunity to actively participate in decision-making processes, including screening and treatment procedures, that affect their health outcome (Silberg et al., 1997Go; Sacchetti et al., 1999Go; Paulter et al., 2001Go). Despite the potential benefit of such interactions, concerns remain regarding the quality of information provided on websites (Jadad and Gagliardi, 1998Go). The deregulated nature of the Internet provides a vehicle for anyone wishing to publish health-related information, evidence-based or anecdotal. Although copyright, libel and privacy laws exist, their interpretation and statutes may often differ between countries (Goudreau, 1999Go). The Internet represents a liberated medium that operates across geographical borders where national laws become increasingly difficult to enforce and have limited jurisdiction. Therefore without governance the quality of online information remains largely uncontrolled and has the potential to negatively influence consumer health outcomes.

In the pursuit of online health information, consumers face the concurrent challenges of finding the information and of determining its quality. Search-engines provide an index of the Internet, allowing an easy and efficient means of accessing and filtering vast amounts of information (Tay et al., 1998Go; Wukovitz, 2001Go). Although hundreds of general search-engines are available, finding relevant and valid health information remains difficult due to the structure and size of the Internet. Specialized medical search-engines have been developed with the aim to make searching easier by identifying websites, according to established criteria, to provide a specific index with medical or healthcare content (Wukovitz, 2001Go). However, even the most efficient search-engine cannot assess the validity and quality of the information, leaving the consumer to evaluate information independently.

Quality of information-rating tools, codes and logos have been developed to provide consumers with the opportunity to identify and appraise the quality of online information (Jadad and Gagliardi, 1998Go). The majority of these tools however, are not specifically designed to evaluate medical information. Most are concerned with the design and presentation of information with few containing validated criteria for quality assessment of that information (Jadad and Gagliardi, 1998Go). Tools such as DISCERN (Charnock et al., 1999Go) and the Health Summit Working Group IQ tool (Health Summit Working Group, 2001Go, accessed on December 11, 2001) have been developed to specifically appraise the quality of medical website information (Charnock et al., 1999Go), allowing the consumer to differentiate between websites containing information they can believe and websites with invalid information.

Although studies assessing Internet use and the quality of information have been performed on various health topics (Impicciatore et al., 1997Go; Graber et al., 1999Go; Latthe et al., 1999Go, 2000; Sacchetti et al., 1999Go; Okamura et al., 2002Go) none has assessed the nature of information regarding androgen deficiency in the ageing male (ADAM). We aimed to compare the search efficiency and the quality of the information available between general search and medical search-engines. We also aimed to determine whether any website characteristics were associated with a higher quality of website information. We chose to assess this health topic (i.e. ADAM) because of the current debate surrounding its physiological validity within the medical community and increasing consumer interest regarding treatment regimes. We also hypothesized that it is a topic that consumers may choose to investigate with the anonymity of the Internet.


    Materials and methods
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
We performed a keyword search on the Internet between November, 2001 and February, 2002 to identify websites providing relevant information about ADAM. This search was performed using four general search-engines (AltaVista, Excite, Google and Yahoo) and five medical search-engines [DrKoop, HealthInsite, Health on the Net (HON), NHSDirect (NHS) and National Library of Medicine (NLM Medline Plus)]. Eighteen keywords, or phrases, and two ‘Boolean searches’ relating to ADAM were used (Table I). Keywords were searched across all nine search-engines while the two Boolean searches were used only when searching the general search-engines, as the medical search-engines did not have the capacity for Boolean logic. The objective of this search strategy was to evaluate the efficiency of the search-engine rather than the online health seeker.


View this table:
[in this window]
[in a new window]
 
Table I. Search terms
 
None of the searches was restricted in the form of language, file format, domain or occurrence in the page. We examined the first 50 websites from each search keyword query because most people rarely examine beyond 50 websites when performing an Internet search (Sacchetti et al., 1999Go). When reviewing search results from Yahoo, only the websites returned from the Yahoo directory were examined because the Yahoo search combines its directory with an index supplied by Google (SearchEngineWatch, 2002Go; accessed May 23, 2002).

Websites were considered to be relevant if they provided sufficient information to answer the question, "How is ADAM recognized as a medical condition and what treatment regimes are available?" In particular we sought information describing the background of the condition, its symptoms, diagnosis and treatment options. Relevant resources included medical or general information websites, online journals and news reports. We excluded websites featuring information on hypogonadism as it is considered a condition independent of ADAM (Petak et al., 2002Go). Websites were excluded from the analysis if they were repeated under a different URL, were a ‘dead’ link, the server was unavailable or had password requirements, and those in a language other than English.

The quality of website information was assessed using the DISCERN quality assessment tool (Charnock et al., 1999Go) The DISCERN quality assessment tool rates the quality of website information on a five point Likert scale (1 = "the publication is ‘poor’ quality and has serious shortcomings and is not a useful or appropriate source of information about treatment choices", 5 = "the publication is ‘good’ quality and is a useful and appropriate source of information about treatment choices") (Charnock, 1998Go). Whilst originally validated to assess written patient information with respect to treatment options, it has since been modified for consumer use in validating health information on the Internet.

Overall search-engine efficiency was measured as the percentage of relevant websites identified by each search-engine from the total number of websites identified for each search query. The efficiency of search-engines in retrieval of quality website information was assessed in addition to the presence and absence of predetermined website characteristics including target audience, authorship and evidence citation to assess if these easily identified criteria were associated with a higher quality of information rating. Analysis of all statistical parameters was performed using a one-tailed Mann–Whitney U-test.


    Results
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
We reviewed 4927 websites of which 52 (1.06%) non-repeated websites were identified as containing information relevant to ADAM. A total of 47 (1.44%) of 3267 and 10 (0.60%) of 1660 websites were located through general and medical search-engines respectively. Five websites were identified on both types of search-engines. We determined that general search-engines were significantly more efficient at retrieving websites relevant to the topic than medical specific search-engines (P = 0.007). Google was identified as the most efficient general search-engine, generating 4.1% relevant websites whilst HON was the most efficient medical search-engine, generating 1.6% relevant websites (Table II).


View this table:
[in this window]
[in a new window]
 
Table II. Number of relevant and total websites identified across general and medical search-engines
 
Of the 52 websites identified, four had a DISCERN score equal >=3, out of a possible 5. Using the DISCERN analysis it was determined that the quality of website information available on ADAM was of a poor standard, the mean (± SD) quality score for all websites being 1.7 ± 0.66. The mean quality scores for websites retrieved on general and medical search-engines were 1.6 ± 0.56 and 2.0 ± 0.94 respectively; however, the quality of website information retrieved using general search-engines did not differ significantly from those retrieved on medical search-engines (P = 0.14).

The current evidence underpinning the condition of ADAM is poor. There are no recognized clinical guidelines rationalizing a standard diagnosis or treatment procedure for ADAM. Despite this lack of evidence, 19 of the 52 websites recommended treatment with testosterone replacement therapy (TRT), two opposed the use of TRT, 24 provided neutral information relaying the benefits and risks associated with TRT and seven recommended alternative therapies. Evidence supporting website recommendations were cited in seven of the 19 websites recommending TRT, seven of the 24 providing neutral information and two of the seven websites recommending alternate therapies. No evidence was cited in the two websites opposing TRT.

The website characteristics identified, which, when present, resulted in a significantly better quality of website information as assessed by DISCERN, included websites with a "target group identified", "links to the website from other websites" and "citation of the evidence" (Table III). Approval of website information from central organizations such as HON attempt to guide consumers to websites with a better quality of information. This study identified that there was no significant difference between websites with external approval and those without (P = 0.21) (Table III).


View this table:
[in this window]
[in a new window]
 
Table III. Parameters of website quality scores relating to presence/absence of website characteristics
 

    Discussion
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
This study demonstrates that although the quality of websites about ADAM identified by general and medical search-engines is similar, general search-engines were significantly more efficient in identifying websites with relevant information compared with medical search-engines. The overall quality of website information relating to ADAM was identified to be poor. Despite the lack of evidence underpinning the condition of ADAM, websites often recommended treatments. However, websites that identified a target readership, were linked from other websites or cited evidence for their information were more likely to contain a significantly better quality of information than websites that did not.

Consumers are turning to the Internet in an attempt to obtain information that will allow them to have a greater role in decision-making processes affecting their health outcomes. Although search-engines are the preferred method of identifying information on the Internet, their use does not guarantee the retrieval of relevant websites. Irrelevant websites often obtain a high ranking due to the manipulation of website characteristics or payment of website placement by authors (Lissman and Boehnlein, 2001Go). Despite the substantial undertaking by commercial and government entities, the creation of medical search-engines does not appear to provide a more specific index of health websites relating to ADAM. Coupled with findings identifying that consumers prefer to use general rather than medical search-engines when searching for medical information (Eysenbach and Kohler, 2002Go), further research is needed to investigate the relative value of medical search-engine development.

Consumers are often presented with conflicting online information, particularly in topics with a poor evidence base. Current tools raise consumer awareness regarding quality issues but do not specifically address the evidence underpinning online information. Invariably consumers are left to their own devices to make an informed decision as these tools lack the power to discriminate between the treatment recommendations of websites and those supported by validated evidence. However, the majority of consumers lack the skills to appraise information critically and identify website characteristics that define the quality of the presented information, hence their reliance on a third party and the continuance of this cycle.

Our results support previous studies that have identified a lack of quality online health information across various topics including reproductive health (Impicciatore et al., 1997Go; Graber et al., 1999Go; Latthe et al., 1999Go, 2000; Okamura et al., 2002Go). This is in contrast with findings by a previous study that identified complete and up to date online information regarding the treatment of erectile dysfunction (Sacchetti et al., 1999Go). Such a contrast in the quality of online information may be representative of a better evidence base highlighting the current disparity of information available within topics concerning male reproductive health. Our findings also lend support to a previous study suggesting that medical search-engines are not as successful in returning relevant medical information in comparison with general search-engines (Graber et al., 1999Go).

Due to the constantly evolving nature of the Internet, these results can only represent a snap shot of the Internet at a given point in time. The medical validity of ADAM is contentious and it is not possible to extrapolate these findings to assume that medical search-engines are inefficient in retrieving relevant websites across all health topics and do not provide an equal or better retrieval of quality website information in other areas. Future studies could investigate whether this trend is apparent in other men’s health issues but should not only examine the quality of website information but whether evidence supports the website treatment recommendations. Future replication of this study could also identify whether a possible change in the medical stance on ADAM existence is reflected in a change in website information and its quality.

Until websites begin to cite evidence-based material, the quality of online information will remain variable, especially in topics such as ADAM in which there is limited evidence. When searching for the best quality online information, sourcing several websites and relying on those identifying a target group, citing references and linking to other websites may increase validity.

In conclusion, although the quality of online information regarding ADAM is shown to be poor, it is apparent that medical search-engines are no better than general search-engines in sourcing for such information. Approval from external bodies does not necessarily assure a better quality of website information whilst current quality assessment tools can only raise consumer awareness regarding website information. Future tools must evaluate whether treatment recommendations are supported by current evidence in the next step to aid consumers in their search for quality online information.


    Acknowledgement
 
This research was funded by the Commonwealth Department of Health and Aging and Andrology Australia, administered through Monash University. Andrology Australia have a website included in the sample identified by this study.


    FOOTNOTES
 
* Professor C.A.Silagy contributed significantly to the design of this project but died prior to its completion. He was Director of the Monash Institute of Health Services Research. Back


    References
 Top
 Abstract
 Introduction
 Materials and methods
 Results
 Discussion
 References
 
Charnock, D. (1998) The DISCERN Handbook. Radcliffe Medical Press Ltd., Abingdon, Oxon, UK.

Charnock, D., Shepperd, S., Needham, G. and Gann, R. (1999) DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. J. Epidemiol. Community Health, 53, 105–111.[Abstract]

Computer Industry Almanac (2002) Available at: http://www.c-i-a.com/pr032102.htm.

Eysenbach, G. and Kohler, C. (2002) How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. Br. Med. J., 324, 573–577.[Abstract/Free Full Text]

Goudreau, K. (1999) The copyright quagmire on the Internet. Comput. Nurs., 17, 82–85.[Medline]

Graber, M., Bergus, G.R. and York, C. (1999) Using the World Wide Web to answer clinical questions: How efficient are different methods of information retrieval? J. Fam. Pract., 48, 520–524.[ISI][Medline]

Health Summit Working Group Information (2001) Available at: http://hitiweb.mitretek.org/iq.

Impicciatore, P., Pandolfini, C., Casella, N. and Bonati, M. (1997) Reliability of health information for the public on the world wide web: Systematic survey of advice on managing fever in children at home. Br. Med. J., 314, 1875–1878.[Abstract/Free Full Text]

Jadad, A. and Gagliardi, A. (1998) Rating health information on the Internet. Navigating to knowledge or to babel? JAMA, 279, 611–614.[Abstract/Free Full Text]

Latthe, M., Latthe, P.M. and Charlton, R. (1999) Quality of information on emergency contraception on the Internet. Brit. J. Fam. Plann., 26, 39–43.[ISI]

Latthe, P., Latthe, M. and Khan, K.S. (2000) Quality of medical information about menorrhagia on the worldwide web. Brit. J. Obstet. Gynecol., 107, 39–43.[ISI]

Lawrence, S. and Giles, CL. (1998) Searching the world wide web. Science, 280, 98–100.[Abstract/Free Full Text]

Lissman, T. and Boehnlein, J.K. (2001) A critical review of internet information about depression. Psychiatr. Serv., 52, 1046–1050.[Abstract/Free Full Text]

Okamura, M., Bernstein, J. and Fidler, A.T. (2002) Assessing the quality of infertility resources on the world wide web: Tools to guide clients through the maze of fact and fiction. J. Midwifery Womens Health, 47, 264–268.[CrossRef][ISI][Medline]

Paulter, S., Tan, J.K., Dugas, G.R., Pus, N., Ferri, M., Hardie, W.R. and Chin, J.L. (2001) Use of the Internet for self-education by patients with prostate cancer. Urology, 57, 230–233.

Petak, S.M., Baskin, H.J., Bergman, D.A., Dickey, R.A. and Nankin, H.R. (2002) ACCE medical guidelines for clinicl practice for the evaluation and treatment of hypogonadism in adult male patients. Endocrine Pract., 8, 439–456.

Sacchetti, P., Zvara, P. and Plante, M.K. (1999) The Internet and patient education – resources and their reliability: Focus on a select urologic topic. Urology, 53, 1117–1120.[CrossRef][ISI][Medline]

SearchEngineWatch (2002) Available at: http://www.searchenginewatch.com/links/major.html.

Silberg, W., Lundberg, G.D. and Musacchio, R.A. (1997) Assessing, controlling, and assuring the quality of medical information on the Internet. JAMA, 277, 1244–1245.[CrossRef][ISI][Medline]

Tay, J., Ke, S. and Lun, K.C. (1998) MediAgent: A WWW-based scalable and self-learning medical search engine. Proc. AMIA Symp., 597–601.

Wukovitz, L. (2001) Using Internet search engines and library catalogs to locate toxicology information. Toxicology, 157, 121–139.[CrossRef][ISI][Medline]

Submitted on September 23, 2002; resubmitted on December 4, 2002; accepted on December 12, 2002.